2023-03-20 10:56:10,948 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b3ae73f5-6ee4-322c-39f8-fb9a5f1d83f7 2023-03-20 10:56:10,962 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.TestAcidGuaranteesWithEagerPolicy timeout: 13 mins 2023-03-20 10:56:10,974 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-03-20 10:56:10,975 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b3ae73f5-6ee4-322c-39f8-fb9a5f1d83f7/cluster_88c58e92-ae5f-fec6-f36b-653ddacb171c, deleteOnExit=true 2023-03-20 10:56:10,975 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-03-20 10:56:10,975 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b3ae73f5-6ee4-322c-39f8-fb9a5f1d83f7/test.cache.data in system properties and HBase conf 2023-03-20 10:56:10,976 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b3ae73f5-6ee4-322c-39f8-fb9a5f1d83f7/hadoop.tmp.dir in system properties and HBase conf 2023-03-20 10:56:10,976 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b3ae73f5-6ee4-322c-39f8-fb9a5f1d83f7/hadoop.log.dir in system properties and HBase conf 2023-03-20 10:56:10,976 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b3ae73f5-6ee4-322c-39f8-fb9a5f1d83f7/mapreduce.cluster.local.dir in system properties and HBase conf 2023-03-20 10:56:10,977 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b3ae73f5-6ee4-322c-39f8-fb9a5f1d83f7/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-03-20 10:56:10,977 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-03-20 10:56:11,085 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-03-20 10:56:11,474 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-03-20 10:56:11,478 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b3ae73f5-6ee4-322c-39f8-fb9a5f1d83f7/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-03-20 10:56:11,479 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b3ae73f5-6ee4-322c-39f8-fb9a5f1d83f7/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-03-20 10:56:11,479 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b3ae73f5-6ee4-322c-39f8-fb9a5f1d83f7/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-03-20 10:56:11,479 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b3ae73f5-6ee4-322c-39f8-fb9a5f1d83f7/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-03-20 10:56:11,480 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b3ae73f5-6ee4-322c-39f8-fb9a5f1d83f7/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-03-20 10:56:11,480 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b3ae73f5-6ee4-322c-39f8-fb9a5f1d83f7/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-03-20 10:56:11,480 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b3ae73f5-6ee4-322c-39f8-fb9a5f1d83f7/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-03-20 10:56:11,481 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b3ae73f5-6ee4-322c-39f8-fb9a5f1d83f7/dfs.journalnode.edits.dir in system properties and HBase conf 2023-03-20 10:56:11,481 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b3ae73f5-6ee4-322c-39f8-fb9a5f1d83f7/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-03-20 10:56:11,481 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b3ae73f5-6ee4-322c-39f8-fb9a5f1d83f7/nfs.dump.dir in system properties and HBase conf 2023-03-20 10:56:11,482 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b3ae73f5-6ee4-322c-39f8-fb9a5f1d83f7/java.io.tmpdir in system properties and HBase conf 2023-03-20 10:56:11,482 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b3ae73f5-6ee4-322c-39f8-fb9a5f1d83f7/dfs.journalnode.edits.dir in system properties and HBase conf 2023-03-20 10:56:11,482 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b3ae73f5-6ee4-322c-39f8-fb9a5f1d83f7/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-03-20 10:56:11,483 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b3ae73f5-6ee4-322c-39f8-fb9a5f1d83f7/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-03-20 10:56:12,131 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-03-20 10:56:12,135 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-03-20 10:56:12,434 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-03-20 10:56:12,643 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-03-20 10:56:12,669 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-03-20 10:56:12,732 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-03-20 10:56:12,793 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b3ae73f5-6ee4-322c-39f8-fb9a5f1d83f7/java.io.tmpdir/Jetty_localhost_localdomain_40485_hdfs____.z30zch/webapp 2023-03-20 10:56:12,957 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:40485 2023-03-20 10:56:13,009 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-03-20 10:56:13,010 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-03-20 10:56:13,456 WARN [Listener at localhost.localdomain/41729] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-03-20 10:56:13,568 WARN [Listener at localhost.localdomain/41729] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-03-20 10:56:13,593 WARN [Listener at localhost.localdomain/41729] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-03-20 10:56:13,601 INFO [Listener at localhost.localdomain/41729] log.Slf4jLog(67): jetty-6.1.26 2023-03-20 10:56:13,609 INFO [Listener at localhost.localdomain/41729] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b3ae73f5-6ee4-322c-39f8-fb9a5f1d83f7/java.io.tmpdir/Jetty_localhost_45403_datanode____.expczn/webapp 2023-03-20 10:56:13,706 INFO [Listener at localhost.localdomain/41729] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:45403 2023-03-20 10:56:14,104 WARN [Listener at localhost.localdomain/43509] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-03-20 10:56:14,807 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xba87faebe21fa3c5: Processing first storage report for DS-b4c9cee3-6057-4e83-9645-d2dcb30551c3 from datanode 20371a92-d140-4bdf-8ccd-77397eedee72 2023-03-20 10:56:14,809 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xba87faebe21fa3c5: from storage DS-b4c9cee3-6057-4e83-9645-d2dcb30551c3 node DatanodeRegistration(127.0.0.1:35541, datanodeUuid=20371a92-d140-4bdf-8ccd-77397eedee72, infoPort=45165, infoSecurePort=0, ipcPort=43509, storageInfo=lv=-57;cid=testClusterID;nsid=1126356010;c=1679309772220), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 2023-03-20 10:56:14,809 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xba87faebe21fa3c5: Processing first storage report for DS-d9f36955-e3f2-4415-8ac6-5580f76db0ff from datanode 20371a92-d140-4bdf-8ccd-77397eedee72 2023-03-20 10:56:14,809 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xba87faebe21fa3c5: from storage DS-d9f36955-e3f2-4415-8ac6-5580f76db0ff node DatanodeRegistration(127.0.0.1:35541, datanodeUuid=20371a92-d140-4bdf-8ccd-77397eedee72, infoPort=45165, infoSecurePort=0, ipcPort=43509, storageInfo=lv=-57;cid=testClusterID;nsid=1126356010;c=1679309772220), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-03-20 10:56:14,883 DEBUG [Listener at localhost.localdomain/43509] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b3ae73f5-6ee4-322c-39f8-fb9a5f1d83f7 2023-03-20 10:56:15,019 INFO [Listener at localhost.localdomain/43509] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b3ae73f5-6ee4-322c-39f8-fb9a5f1d83f7/cluster_88c58e92-ae5f-fec6-f36b-653ddacb171c/zookeeper_0, clientPort=60226, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b3ae73f5-6ee4-322c-39f8-fb9a5f1d83f7/cluster_88c58e92-ae5f-fec6-f36b-653ddacb171c/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b3ae73f5-6ee4-322c-39f8-fb9a5f1d83f7/cluster_88c58e92-ae5f-fec6-f36b-653ddacb171c/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-03-20 10:56:15,041 INFO [Listener at localhost.localdomain/43509] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=60226 2023-03-20 10:56:15,052 INFO [Listener at localhost.localdomain/43509] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-20 10:56:15,055 INFO [Listener at localhost.localdomain/43509] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-20 10:56:15,763 INFO [Listener at localhost.localdomain/43509] util.FSUtils(471): Created version file at hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a with version=8 2023-03-20 10:56:15,763 INFO [Listener at localhost.localdomain/43509] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/hbase-staging 2023-03-20 10:56:16,049 INFO [Listener at localhost.localdomain/43509] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-03-20 10:56:16,454 INFO [Listener at localhost.localdomain/43509] client.ConnectionUtils(127): master/jenkins-hbase20:0 server-side Connection retries=45 2023-03-20 10:56:16,480 INFO [Listener at localhost.localdomain/43509] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-20 10:56:16,480 INFO [Listener at localhost.localdomain/43509] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-03-20 10:56:16,480 INFO [Listener at localhost.localdomain/43509] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-03-20 10:56:16,481 INFO [Listener at localhost.localdomain/43509] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-20 10:56:16,481 INFO [Listener at localhost.localdomain/43509] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-03-20 10:56:16,602 INFO [Listener at localhost.localdomain/43509] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-03-20 10:56:16,669 DEBUG [Listener at localhost.localdomain/43509] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-03-20 10:56:16,780 INFO [Listener at localhost.localdomain/43509] ipc.NettyRpcServer(120): Bind to /148.251.75.209:44257 2023-03-20 10:56:16,794 INFO [Listener at localhost.localdomain/43509] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-20 10:56:16,796 INFO [Listener at localhost.localdomain/43509] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-20 10:56:16,820 INFO [Listener at localhost.localdomain/43509] zookeeper.RecoverableZooKeeper(93): Process identifier=master:44257 connecting to ZooKeeper ensemble=127.0.0.1:60226 2023-03-20 10:56:16,877 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:442570x0, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-03-20 10:56:16,886 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:44257-0x1002e5fb8b80000 connected 2023-03-20 10:56:16,936 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-03-20 10:56:16,937 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-20 10:56:16,941 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-03-20 10:56:16,954 DEBUG [Listener at localhost.localdomain/43509] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=44257 2023-03-20 10:56:16,955 DEBUG [Listener at localhost.localdomain/43509] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=44257 2023-03-20 10:56:16,962 DEBUG [Listener at localhost.localdomain/43509] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=44257 2023-03-20 10:56:16,970 DEBUG [Listener at localhost.localdomain/43509] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=44257 2023-03-20 10:56:16,974 DEBUG [Listener at localhost.localdomain/43509] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=44257 2023-03-20 10:56:16,982 INFO [Listener at localhost.localdomain/43509] master.HMaster(444): hbase.rootdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a, hbase.cluster.distributed=false 2023-03-20 10:56:17,063 INFO [Listener at localhost.localdomain/43509] client.ConnectionUtils(127): regionserver/jenkins-hbase20:0 server-side Connection retries=45 2023-03-20 10:56:17,063 INFO [Listener at localhost.localdomain/43509] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-20 10:56:17,064 INFO [Listener at localhost.localdomain/43509] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-03-20 10:56:17,064 INFO [Listener at localhost.localdomain/43509] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-03-20 10:56:17,064 INFO [Listener at localhost.localdomain/43509] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-20 10:56:17,064 INFO [Listener at localhost.localdomain/43509] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-03-20 10:56:17,069 INFO [Listener at localhost.localdomain/43509] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-03-20 10:56:17,072 INFO [Listener at localhost.localdomain/43509] ipc.NettyRpcServer(120): Bind to /148.251.75.209:36041 2023-03-20 10:56:17,074 INFO [Listener at localhost.localdomain/43509] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-03-20 10:56:17,080 DEBUG [Listener at localhost.localdomain/43509] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-03-20 10:56:17,082 INFO [Listener at localhost.localdomain/43509] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-20 10:56:17,086 INFO [Listener at localhost.localdomain/43509] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-20 10:56:17,089 INFO [Listener at localhost.localdomain/43509] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:36041 connecting to ZooKeeper ensemble=127.0.0.1:60226 2023-03-20 10:56:17,092 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:360410x0, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-03-20 10:56:17,093 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:36041-0x1002e5fb8b80001 connected 2023-03-20 10:56:17,093 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-03-20 10:56:17,094 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-20 10:56:17,095 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-03-20 10:56:17,096 DEBUG [Listener at localhost.localdomain/43509] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=36041 2023-03-20 10:56:17,096 DEBUG [Listener at localhost.localdomain/43509] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=36041 2023-03-20 10:56:17,096 DEBUG [Listener at localhost.localdomain/43509] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=36041 2023-03-20 10:56:17,097 DEBUG [Listener at localhost.localdomain/43509] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=36041 2023-03-20 10:56:17,097 DEBUG [Listener at localhost.localdomain/43509] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=36041 2023-03-20 10:56:17,099 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase20.apache.org,44257,1679309775907 2023-03-20 10:56:17,107 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-03-20 10:56:17,109 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase20.apache.org,44257,1679309775907 2023-03-20 10:56:17,126 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-03-20 10:56:17,126 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-03-20 10:56:17,126 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-20 10:56:17,127 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-03-20 10:56:17,128 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase20.apache.org,44257,1679309775907 from backup master directory 2023-03-20 10:56:17,128 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-03-20 10:56:17,131 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase20.apache.org,44257,1679309775907 2023-03-20 10:56:17,131 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-03-20 10:56:17,132 WARN [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-03-20 10:56:17,132 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase20.apache.org,44257,1679309775907 2023-03-20 10:56:17,135 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-03-20 10:56:17,136 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-03-20 10:56:17,617 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/hbase.id with ID: 9a9d3dd3-cde9-41c6-84a9-10f20cd971d6 2023-03-20 10:56:17,662 INFO [master/jenkins-hbase20:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-20 10:56:17,675 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-20 10:56:18,110 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x58fe3863 to 127.0.0.1:60226 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 10:56:18,150 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@64c55a3a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 10:56:18,172 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-20 10:56:18,173 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-03-20 10:56:18,189 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-03-20 10:56:18,189 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-03-20 10:56:18,191 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-03-20 10:56:18,195 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-03-20 10:56:18,196 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-03-20 10:56:18,226 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/MasterData/data/master/store-tmp 2023-03-20 10:56:18,253 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-20 10:56:18,253 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-03-20 10:56:18,253 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-20 10:56:18,253 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-20 10:56:18,253 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-03-20 10:56:18,253 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-20 10:56:18,254 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-20 10:56:18,254 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-03-20 10:56:18,256 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/MasterData/WALs/jenkins-hbase20.apache.org,44257,1679309775907 2023-03-20 10:56:18,281 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C44257%2C1679309775907, suffix=, logDir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/MasterData/WALs/jenkins-hbase20.apache.org,44257,1679309775907, archiveDir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/MasterData/oldWALs, maxLogs=10 2023-03-20 10:56:18,350 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35541,DS-b4c9cee3-6057-4e83-9645-d2dcb30551c3,DISK] 2023-03-20 10:56:18,357 DEBUG [RS-EventLoopGroup-3-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-20 10:56:18,412 INFO [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/MasterData/WALs/jenkins-hbase20.apache.org,44257,1679309775907/jenkins-hbase20.apache.org%2C44257%2C1679309775907.1679309778294 2023-03-20 10:56:18,413 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:35541,DS-b4c9cee3-6057-4e83-9645-d2dcb30551c3,DISK]] 2023-03-20 10:56:18,414 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-03-20 10:56:18,414 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-20 10:56:18,417 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-03-20 10:56:18,418 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-03-20 10:56:18,479 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-03-20 10:56:18,487 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-03-20 10:56:18,518 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-03-20 10:56:18,529 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-20 10:56:18,534 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-03-20 10:56:18,536 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-03-20 10:56:18,553 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-03-20 10:56:18,558 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-20 10:56:18,559 INFO [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60225198, jitterRate=-0.10257461667060852}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-20 10:56:18,559 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-03-20 10:56:18,560 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-03-20 10:56:18,587 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-03-20 10:56:18,587 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-03-20 10:56:18,589 INFO [master/jenkins-hbase20:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-03-20 10:56:18,591 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-03-20 10:56:18,621 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 29 msec 2023-03-20 10:56:18,621 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-03-20 10:56:18,645 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-03-20 10:56:18,651 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-03-20 10:56:18,672 INFO [master/jenkins-hbase20:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-03-20 10:56:18,675 INFO [master/jenkins-hbase20:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-03-20 10:56:18,677 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-03-20 10:56:18,681 INFO [master/jenkins-hbase20:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-03-20 10:56:18,685 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-03-20 10:56:18,687 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-20 10:56:18,688 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-03-20 10:56:18,689 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-03-20 10:56:18,700 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-03-20 10:56:18,704 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-03-20 10:56:18,704 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-03-20 10:56:18,704 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-20 10:56:18,705 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase20.apache.org,44257,1679309775907, sessionid=0x1002e5fb8b80000, setting cluster-up flag (Was=false) 2023-03-20 10:56:18,721 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-20 10:56:18,725 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-03-20 10:56:18,727 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase20.apache.org,44257,1679309775907 2023-03-20 10:56:18,746 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-20 10:56:18,752 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-03-20 10:56:18,753 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase20.apache.org,44257,1679309775907 2023-03-20 10:56:18,756 WARN [master/jenkins-hbase20:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.hbase-snapshot/.tmp 2023-03-20 10:56:18,802 INFO [RS:0;jenkins-hbase20:36041] regionserver.HRegionServer(952): ClusterId : 9a9d3dd3-cde9-41c6-84a9-10f20cd971d6 2023-03-20 10:56:18,807 DEBUG [RS:0;jenkins-hbase20:36041] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-03-20 10:56:18,813 DEBUG [RS:0;jenkins-hbase20:36041] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-03-20 10:56:18,813 DEBUG [RS:0;jenkins-hbase20:36041] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-03-20 10:56:18,817 DEBUG [RS:0;jenkins-hbase20:36041] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-03-20 10:56:18,818 DEBUG [RS:0;jenkins-hbase20:36041] zookeeper.ReadOnlyZKClient(139): Connect 0x57f52db3 to 127.0.0.1:60226 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 10:56:18,831 DEBUG [RS:0;jenkins-hbase20:36041] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5932d93d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 10:56:18,832 DEBUG [RS:0;jenkins-hbase20:36041] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5d0d46c3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-03-20 10:56:18,857 DEBUG [RS:0;jenkins-hbase20:36041] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase20:36041 2023-03-20 10:56:18,862 INFO [RS:0;jenkins-hbase20:36041] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-03-20 10:56:18,862 INFO [RS:0;jenkins-hbase20:36041] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-03-20 10:56:18,862 DEBUG [RS:0;jenkins-hbase20:36041] regionserver.HRegionServer(1023): About to register with Master. 2023-03-20 10:56:18,865 INFO [RS:0;jenkins-hbase20:36041] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase20.apache.org,44257,1679309775907 with isa=jenkins-hbase20.apache.org/148.251.75.209:36041, startcode=1679309777062 2023-03-20 10:56:18,885 DEBUG [RS:0;jenkins-hbase20:36041] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-03-20 10:56:18,885 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-03-20 10:56:18,897 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-03-20 10:56:18,897 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-03-20 10:56:18,898 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-03-20 10:56:18,898 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=5, maxPoolSize=5 2023-03-20 10:56:18,898 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase20:0, corePoolSize=10, maxPoolSize=10 2023-03-20 10:56:18,898 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-03-20 10:56:18,898 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-03-20 10:56:18,898 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-03-20 10:56:18,906 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1679309808906 2023-03-20 10:56:18,909 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-03-20 10:56:18,912 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-03-20 10:56:18,912 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-03-20 10:56:18,919 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-03-20 10:56:18,922 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-03-20 10:56:18,931 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-03-20 10:56:18,932 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-03-20 10:56:18,932 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-03-20 10:56:18,932 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-03-20 10:56:18,934 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-03-20 10:56:18,936 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-03-20 10:56:18,938 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-03-20 10:56:18,939 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-03-20 10:56:18,942 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-03-20 10:56:18,942 INFO [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-03-20 10:56:18,945 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1679309778944,5,FailOnTimeoutGroup] 2023-03-20 10:56:18,945 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1679309778945,5,FailOnTimeoutGroup] 2023-03-20 10:56:18,945 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-03-20 10:56:18,945 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-03-20 10:56:18,947 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:51029, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-03-20 10:56:18,947 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-03-20 10:56:18,948 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-03-20 10:56:18,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.ServerManager(394): Registering regionserver=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:18,975 DEBUG [RS:0;jenkins-hbase20:36041] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a 2023-03-20 10:56:18,975 DEBUG [RS:0;jenkins-hbase20:36041] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost.localdomain:41729 2023-03-20 10:56:18,975 DEBUG [RS:0;jenkins-hbase20:36041] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-03-20 10:56:18,980 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-03-20 10:56:18,980 DEBUG [RS:0;jenkins-hbase20:36041] zookeeper.ZKUtil(162): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:18,981 WARN [RS:0;jenkins-hbase20:36041] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-03-20 10:56:18,981 INFO [RS:0;jenkins-hbase20:36041] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-03-20 10:56:18,982 DEBUG [RS:0;jenkins-hbase20:36041] regionserver.HRegionServer(1947): logDir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/WALs/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:18,983 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase20.apache.org,36041,1679309777062] 2023-03-20 10:56:18,992 DEBUG [RS:0;jenkins-hbase20:36041] zookeeper.ZKUtil(162): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:19,005 DEBUG [RS:0;jenkins-hbase20:36041] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-03-20 10:56:19,014 INFO [RS:0;jenkins-hbase20:36041] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-03-20 10:56:19,036 INFO [RS:0;jenkins-hbase20:36041] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-03-20 10:56:19,040 INFO [RS:0;jenkins-hbase20:36041] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-03-20 10:56:19,040 INFO [RS:0;jenkins-hbase20:36041] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-03-20 10:56:19,042 INFO [RS:0;jenkins-hbase20:36041] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-03-20 10:56:19,050 INFO [RS:0;jenkins-hbase20:36041] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-03-20 10:56:19,051 DEBUG [RS:0;jenkins-hbase20:36041] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-03-20 10:56:19,051 DEBUG [RS:0;jenkins-hbase20:36041] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-03-20 10:56:19,051 DEBUG [RS:0;jenkins-hbase20:36041] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-03-20 10:56:19,052 DEBUG [RS:0;jenkins-hbase20:36041] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-03-20 10:56:19,052 DEBUG [RS:0;jenkins-hbase20:36041] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-03-20 10:56:19,052 DEBUG [RS:0;jenkins-hbase20:36041] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase20:0, corePoolSize=2, maxPoolSize=2 2023-03-20 10:56:19,052 DEBUG [RS:0;jenkins-hbase20:36041] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-03-20 10:56:19,052 DEBUG [RS:0;jenkins-hbase20:36041] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-03-20 10:56:19,053 DEBUG [RS:0;jenkins-hbase20:36041] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-03-20 10:56:19,053 DEBUG [RS:0;jenkins-hbase20:36041] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase20:0, corePoolSize=1, maxPoolSize=1 2023-03-20 10:56:19,055 INFO [RS:0;jenkins-hbase20:36041] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-03-20 10:56:19,056 INFO [RS:0;jenkins-hbase20:36041] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-03-20 10:56:19,056 INFO [RS:0;jenkins-hbase20:36041] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-03-20 10:56:19,069 INFO [RS:0;jenkins-hbase20:36041] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-03-20 10:56:19,071 INFO [RS:0;jenkins-hbase20:36041] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,36041,1679309777062-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-03-20 10:56:19,086 INFO [RS:0;jenkins-hbase20:36041] regionserver.Replication(203): jenkins-hbase20.apache.org,36041,1679309777062 started 2023-03-20 10:56:19,086 INFO [RS:0;jenkins-hbase20:36041] regionserver.HRegionServer(1638): Serving as jenkins-hbase20.apache.org,36041,1679309777062, RpcServer on jenkins-hbase20.apache.org/148.251.75.209:36041, sessionid=0x1002e5fb8b80001 2023-03-20 10:56:19,086 DEBUG [RS:0;jenkins-hbase20:36041] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-03-20 10:56:19,086 DEBUG [RS:0;jenkins-hbase20:36041] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:19,086 DEBUG [RS:0;jenkins-hbase20:36041] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,36041,1679309777062' 2023-03-20 10:56:19,086 DEBUG [RS:0;jenkins-hbase20:36041] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:19,087 DEBUG [RS:0;jenkins-hbase20:36041] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:19,088 DEBUG [RS:0;jenkins-hbase20:36041] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-03-20 10:56:19,088 DEBUG [RS:0;jenkins-hbase20:36041] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-03-20 10:56:19,088 DEBUG [RS:0;jenkins-hbase20:36041] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:19,088 DEBUG [RS:0;jenkins-hbase20:36041] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase20.apache.org,36041,1679309777062' 2023-03-20 10:56:19,088 DEBUG [RS:0;jenkins-hbase20:36041] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-03-20 10:56:19,088 DEBUG [RS:0;jenkins-hbase20:36041] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-03-20 10:56:19,089 DEBUG [RS:0;jenkins-hbase20:36041] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-03-20 10:56:19,089 INFO [RS:0;jenkins-hbase20:36041] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-03-20 10:56:19,089 INFO [RS:0;jenkins-hbase20:36041] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-03-20 10:56:19,199 INFO [RS:0;jenkins-hbase20:36041] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C36041%2C1679309777062, suffix=, logDir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/WALs/jenkins-hbase20.apache.org,36041,1679309777062, archiveDir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/oldWALs, maxLogs=32 2023-03-20 10:56:19,217 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35541,DS-b4c9cee3-6057-4e83-9645-d2dcb30551c3,DISK] 2023-03-20 10:56:19,222 INFO [RS:0;jenkins-hbase20:36041] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/WALs/jenkins-hbase20.apache.org,36041,1679309777062/jenkins-hbase20.apache.org%2C36041%2C1679309777062.1679309779202 2023-03-20 10:56:19,223 DEBUG [RS:0;jenkins-hbase20:36041] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:35541,DS-b4c9cee3-6057-4e83-9645-d2dcb30551c3,DISK]] 2023-03-20 10:56:19,361 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-03-20 10:56:19,363 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-03-20 10:56:19,363 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a 2023-03-20 10:56:19,781 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-20 10:56:19,785 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-03-20 10:56:19,791 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/hbase/meta/1588230740/info 2023-03-20 10:56:19,792 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-03-20 10:56:19,794 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-20 10:56:19,794 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-03-20 10:56:19,798 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/hbase/meta/1588230740/rep_barrier 2023-03-20 10:56:19,799 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-03-20 10:56:19,801 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-20 10:56:19,801 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-03-20 10:56:19,805 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/hbase/meta/1588230740/table 2023-03-20 10:56:19,805 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-03-20 10:56:19,806 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-20 10:56:19,807 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/hbase/meta/1588230740 2023-03-20 10:56:19,808 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/hbase/meta/1588230740 2023-03-20 10:56:19,811 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-20 10:56:19,812 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-03-20 10:56:19,815 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-20 10:56:19,816 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73830759, jitterRate=0.10016404092311859}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-20 10:56:19,816 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-03-20 10:56:19,816 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-03-20 10:56:19,816 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-03-20 10:56:19,816 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-03-20 10:56:19,816 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-03-20 10:56:19,817 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-03-20 10:56:19,817 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-03-20 10:56:19,818 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-03-20 10:56:19,823 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-03-20 10:56:19,823 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-03-20 10:56:19,830 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-03-20 10:56:19,843 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-03-20 10:56:19,845 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-03-20 10:56:19,999 DEBUG [jenkins-hbase20:44257] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-03-20 10:56:20,004 INFO [PEWorker-4] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase20.apache.org,36041,1679309777062, state=OPENING 2023-03-20 10:56:20,013 DEBUG [PEWorker-4] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-03-20 10:56:20,015 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-20 10:56:20,015 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-03-20 10:56:20,017 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase20.apache.org,36041,1679309777062}] 2023-03-20 10:56:20,198 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:20,200 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-03-20 10:56:20,204 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:59486, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-03-20 10:56:20,215 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-03-20 10:56:20,215 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-03-20 10:56:20,218 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase20.apache.org%2C36041%2C1679309777062.meta, suffix=.meta, logDir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/WALs/jenkins-hbase20.apache.org,36041,1679309777062, archiveDir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/oldWALs, maxLogs=32 2023-03-20 10:56:20,234 DEBUG [RS-EventLoopGroup-3-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35541,DS-b4c9cee3-6057-4e83-9645-d2dcb30551c3,DISK] 2023-03-20 10:56:20,238 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/WALs/jenkins-hbase20.apache.org,36041,1679309777062/jenkins-hbase20.apache.org%2C36041%2C1679309777062.meta.1679309780220.meta 2023-03-20 10:56:20,239 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:35541,DS-b4c9cee3-6057-4e83-9645-d2dcb30551c3,DISK]] 2023-03-20 10:56:20,239 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-03-20 10:56:20,241 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-03-20 10:56:20,256 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-03-20 10:56:20,261 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-03-20 10:56:20,268 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-03-20 10:56:20,268 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-20 10:56:20,268 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-03-20 10:56:20,268 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-03-20 10:56:20,276 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-03-20 10:56:20,278 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/hbase/meta/1588230740/info 2023-03-20 10:56:20,278 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/hbase/meta/1588230740/info 2023-03-20 10:56:20,279 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-03-20 10:56:20,280 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-20 10:56:20,281 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-03-20 10:56:20,282 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/hbase/meta/1588230740/rep_barrier 2023-03-20 10:56:20,283 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/hbase/meta/1588230740/rep_barrier 2023-03-20 10:56:20,283 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-03-20 10:56:20,285 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-20 10:56:20,285 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-03-20 10:56:20,287 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/hbase/meta/1588230740/table 2023-03-20 10:56:20,287 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/hbase/meta/1588230740/table 2023-03-20 10:56:20,288 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-03-20 10:56:20,289 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-20 10:56:20,291 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/hbase/meta/1588230740 2023-03-20 10:56:20,295 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/hbase/meta/1588230740 2023-03-20 10:56:20,300 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-20 10:56:20,304 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-03-20 10:56:20,312 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74184207, jitterRate=0.10543082654476166}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-20 10:56:20,312 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-03-20 10:56:20,324 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1679309780189 2023-03-20 10:56:20,343 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-03-20 10:56:20,344 INFO [RS_OPEN_META-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-03-20 10:56:20,344 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase20.apache.org,36041,1679309777062, state=OPEN 2023-03-20 10:56:20,347 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-03-20 10:56:20,347 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-03-20 10:56:20,351 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-03-20 10:56:20,351 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase20.apache.org,36041,1679309777062 in 330 msec 2023-03-20 10:56:20,356 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-03-20 10:56:20,356 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 522 msec 2023-03-20 10:56:20,363 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 1.5610 sec 2023-03-20 10:56:20,363 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1679309780363, completionTime=-1 2023-03-20 10:56:20,363 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-03-20 10:56:20,364 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-03-20 10:56:20,420 DEBUG [hconnection-0x663d22de-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 10:56:20,423 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:59496, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 10:56:20,444 INFO [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-03-20 10:56:20,445 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1679309840444 2023-03-20 10:56:20,445 INFO [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1679309900445 2023-03-20 10:56:20,445 INFO [master/jenkins-hbase20:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 81 msec 2023-03-20 10:56:20,472 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,44257,1679309775907-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-03-20 10:56:20,473 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,44257,1679309775907-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-03-20 10:56:20,473 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,44257,1679309775907-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-03-20 10:56:20,475 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase20:44257, period=300000, unit=MILLISECONDS is enabled. 2023-03-20 10:56:20,476 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-03-20 10:56:20,484 DEBUG [master/jenkins-hbase20:0.Chore.1] janitor.CatalogJanitor(175): 2023-03-20 10:56:20,494 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-03-20 10:56:20,496 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-03-20 10:56:20,507 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-03-20 10:56:20,518 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-03-20 10:56:20,522 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-20 10:56:20,544 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/hbase/namespace/39b9d351fb81939da79179eb526f8a01 2023-03-20 10:56:20,547 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/hbase/namespace/39b9d351fb81939da79179eb526f8a01 empty. 2023-03-20 10:56:20,547 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/hbase/namespace/39b9d351fb81939da79179eb526f8a01 2023-03-20 10:56:20,547 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-03-20 10:56:20,569 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-03-20 10:56:20,572 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 39b9d351fb81939da79179eb526f8a01, NAME => 'hbase:namespace,,1679309780495.39b9d351fb81939da79179eb526f8a01.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp 2023-03-20 10:56:20,588 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1679309780495.39b9d351fb81939da79179eb526f8a01.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-20 10:56:20,589 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 39b9d351fb81939da79179eb526f8a01, disabling compactions & flushes 2023-03-20 10:56:20,589 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1679309780495.39b9d351fb81939da79179eb526f8a01. 2023-03-20 10:56:20,589 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1679309780495.39b9d351fb81939da79179eb526f8a01. 2023-03-20 10:56:20,589 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1679309780495.39b9d351fb81939da79179eb526f8a01. after waiting 0 ms 2023-03-20 10:56:20,589 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1679309780495.39b9d351fb81939da79179eb526f8a01. 2023-03-20 10:56:20,589 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1679309780495.39b9d351fb81939da79179eb526f8a01. 2023-03-20 10:56:20,589 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 39b9d351fb81939da79179eb526f8a01: 2023-03-20 10:56:20,594 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-03-20 10:56:20,612 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1679309780495.39b9d351fb81939da79179eb526f8a01.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1679309780598"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679309780598"}]},"ts":"1679309780598"} 2023-03-20 10:56:20,649 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-20 10:56:20,652 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-20 10:56:20,658 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679309780652"}]},"ts":"1679309780652"} 2023-03-20 10:56:20,667 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-03-20 10:56:20,675 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=39b9d351fb81939da79179eb526f8a01, ASSIGN}] 2023-03-20 10:56:20,679 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=39b9d351fb81939da79179eb526f8a01, ASSIGN 2023-03-20 10:56:20,681 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=39b9d351fb81939da79179eb526f8a01, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,36041,1679309777062; forceNewPlan=false, retain=false 2023-03-20 10:56:20,832 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=39b9d351fb81939da79179eb526f8a01, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:20,833 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1679309780495.39b9d351fb81939da79179eb526f8a01.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1679309780832"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679309780832"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679309780832"}]},"ts":"1679309780832"} 2023-03-20 10:56:20,837 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 39b9d351fb81939da79179eb526f8a01, server=jenkins-hbase20.apache.org,36041,1679309777062}] 2023-03-20 10:56:21,003 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1679309780495.39b9d351fb81939da79179eb526f8a01. 2023-03-20 10:56:21,004 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 39b9d351fb81939da79179eb526f8a01, NAME => 'hbase:namespace,,1679309780495.39b9d351fb81939da79179eb526f8a01.', STARTKEY => '', ENDKEY => ''} 2023-03-20 10:56:21,005 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 39b9d351fb81939da79179eb526f8a01 2023-03-20 10:56:21,006 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1679309780495.39b9d351fb81939da79179eb526f8a01.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-20 10:56:21,006 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 39b9d351fb81939da79179eb526f8a01 2023-03-20 10:56:21,006 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 39b9d351fb81939da79179eb526f8a01 2023-03-20 10:56:21,010 INFO [StoreOpener-39b9d351fb81939da79179eb526f8a01-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 39b9d351fb81939da79179eb526f8a01 2023-03-20 10:56:21,014 DEBUG [StoreOpener-39b9d351fb81939da79179eb526f8a01-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/hbase/namespace/39b9d351fb81939da79179eb526f8a01/info 2023-03-20 10:56:21,014 DEBUG [StoreOpener-39b9d351fb81939da79179eb526f8a01-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/hbase/namespace/39b9d351fb81939da79179eb526f8a01/info 2023-03-20 10:56:21,015 INFO [StoreOpener-39b9d351fb81939da79179eb526f8a01-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 39b9d351fb81939da79179eb526f8a01 columnFamilyName info 2023-03-20 10:56:21,016 INFO [StoreOpener-39b9d351fb81939da79179eb526f8a01-1] regionserver.HStore(310): Store=39b9d351fb81939da79179eb526f8a01/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-20 10:56:21,018 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/hbase/namespace/39b9d351fb81939da79179eb526f8a01 2023-03-20 10:56:21,019 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/hbase/namespace/39b9d351fb81939da79179eb526f8a01 2023-03-20 10:56:21,025 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 39b9d351fb81939da79179eb526f8a01 2023-03-20 10:56:21,030 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/hbase/namespace/39b9d351fb81939da79179eb526f8a01/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-20 10:56:21,031 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 39b9d351fb81939da79179eb526f8a01; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=58866844, jitterRate=-0.12281566858291626}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-20 10:56:21,031 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 39b9d351fb81939da79179eb526f8a01: 2023-03-20 10:56:21,034 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1679309780495.39b9d351fb81939da79179eb526f8a01., pid=6, masterSystemTime=1679309780992 2023-03-20 10:56:21,041 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1679309780495.39b9d351fb81939da79179eb526f8a01. 2023-03-20 10:56:21,042 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1679309780495.39b9d351fb81939da79179eb526f8a01. 2023-03-20 10:56:21,044 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=39b9d351fb81939da79179eb526f8a01, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:21,045 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1679309780495.39b9d351fb81939da79179eb526f8a01.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1679309781043"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1679309781043"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679309781043"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679309781043"}]},"ts":"1679309781043"} 2023-03-20 10:56:21,055 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-03-20 10:56:21,055 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 39b9d351fb81939da79179eb526f8a01, server=jenkins-hbase20.apache.org,36041,1679309777062 in 213 msec 2023-03-20 10:56:21,061 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-03-20 10:56:21,062 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=39b9d351fb81939da79179eb526f8a01, ASSIGN in 380 msec 2023-03-20 10:56:21,064 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-20 10:56:21,065 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679309781064"}]},"ts":"1679309781064"} 2023-03-20 10:56:21,069 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-03-20 10:56:21,074 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-03-20 10:56:21,079 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 578 msec 2023-03-20 10:56:21,111 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-03-20 10:56:21,112 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-03-20 10:56:21,112 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-20 10:56:21,153 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-03-20 10:56:21,174 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-03-20 10:56:21,184 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 38 msec 2023-03-20 10:56:21,190 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-03-20 10:56:21,213 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-03-20 10:56:21,218 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 28 msec 2023-03-20 10:56:21,228 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-03-20 10:56:21,230 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-03-20 10:56:21,232 INFO [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 4.100sec 2023-03-20 10:56:21,234 INFO [master/jenkins-hbase20:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-03-20 10:56:21,236 INFO [master/jenkins-hbase20:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-03-20 10:56:21,236 INFO [master/jenkins-hbase20:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-03-20 10:56:21,237 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,44257,1679309775907-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-03-20 10:56:21,238 INFO [master/jenkins-hbase20:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase20.apache.org,44257,1679309775907-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-03-20 10:56:21,251 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-03-20 10:56:21,309 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(139): Connect 0x70531455 to 127.0.0.1:60226 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 10:56:21,317 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6a972b34, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 10:56:21,333 DEBUG [hconnection-0x5e53e335-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 10:56:21,347 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41352, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 10:56:21,356 INFO [Listener at localhost.localdomain/43509] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase20.apache.org,44257,1679309775907 2023-03-20 10:56:21,374 INFO [Listener at localhost.localdomain/43509] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=205, OpenFileDescriptor=470, MaxFileDescriptor=60000, SystemLoadAverage=417, ProcessCount=184, AvailableMemoryMB=9684 2023-03-20 10:56:21,389 DEBUG [Listener at localhost.localdomain/43509] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-20 10:56:21,392 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:47918, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-20 10:56:21,403 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-20 10:56:21,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-20 10:56:21,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-20 10:56:21,413 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-20 10:56:21,416 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-20 10:56:21,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 9 2023-03-20 10:56:21,419 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1 2023-03-20 10:56:21,421 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1 empty. 2023-03-20 10:56:21,423 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1 2023-03-20 10:56:21,423 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-20 10:56:21,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-20 10:56:21,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-20 10:56:21,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-20 10:56:21,846 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-20 10:56:21,850 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => c79c35ba58052cacab6709e8682a60f1, NAME => 'TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp 2023-03-20 10:56:21,862 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-20 10:56:21,862 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing c79c35ba58052cacab6709e8682a60f1, disabling compactions & flushes 2023-03-20 10:56:21,863 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:21,863 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:21,863 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. after waiting 0 ms 2023-03-20 10:56:21,863 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:21,863 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:21,863 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for c79c35ba58052cacab6709e8682a60f1: 2023-03-20 10:56:21,869 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-20 10:56:21,871 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679309781871"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679309781871"}]},"ts":"1679309781871"} 2023-03-20 10:56:21,874 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-20 10:56:21,875 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-20 10:56:21,876 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679309781876"}]},"ts":"1679309781876"} 2023-03-20 10:56:21,878 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-20 10:56:21,881 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c79c35ba58052cacab6709e8682a60f1, ASSIGN}] 2023-03-20 10:56:21,883 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c79c35ba58052cacab6709e8682a60f1, ASSIGN 2023-03-20 10:56:21,885 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=c79c35ba58052cacab6709e8682a60f1, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,36041,1679309777062; forceNewPlan=false, retain=false 2023-03-20 10:56:22,036 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=c79c35ba58052cacab6709e8682a60f1, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:22,036 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679309782036"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679309782036"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679309782036"}]},"ts":"1679309782036"} 2023-03-20 10:56:22,040 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062}] 2023-03-20 10:56:22,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-20 10:56:22,200 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:22,200 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => c79c35ba58052cacab6709e8682a60f1, NAME => 'TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.', STARTKEY => '', ENDKEY => ''} 2023-03-20 10:56:22,201 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees c79c35ba58052cacab6709e8682a60f1 2023-03-20 10:56:22,201 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-20 10:56:22,201 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for c79c35ba58052cacab6709e8682a60f1 2023-03-20 10:56:22,201 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for c79c35ba58052cacab6709e8682a60f1 2023-03-20 10:56:22,203 INFO [StoreOpener-c79c35ba58052cacab6709e8682a60f1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region c79c35ba58052cacab6709e8682a60f1 2023-03-20 10:56:22,206 DEBUG [StoreOpener-c79c35ba58052cacab6709e8682a60f1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A 2023-03-20 10:56:22,206 DEBUG [StoreOpener-c79c35ba58052cacab6709e8682a60f1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A 2023-03-20 10:56:22,207 INFO [StoreOpener-c79c35ba58052cacab6709e8682a60f1-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-20 10:56:22,208 INFO [StoreOpener-c79c35ba58052cacab6709e8682a60f1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c79c35ba58052cacab6709e8682a60f1 columnFamilyName A 2023-03-20 10:56:22,209 INFO [StoreOpener-c79c35ba58052cacab6709e8682a60f1-1] regionserver.HStore(310): Store=c79c35ba58052cacab6709e8682a60f1/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-20 10:56:22,209 INFO [StoreOpener-c79c35ba58052cacab6709e8682a60f1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region c79c35ba58052cacab6709e8682a60f1 2023-03-20 10:56:22,212 DEBUG [StoreOpener-c79c35ba58052cacab6709e8682a60f1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B 2023-03-20 10:56:22,212 DEBUG [StoreOpener-c79c35ba58052cacab6709e8682a60f1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B 2023-03-20 10:56:22,212 INFO [StoreOpener-c79c35ba58052cacab6709e8682a60f1-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-20 10:56:22,213 INFO [StoreOpener-c79c35ba58052cacab6709e8682a60f1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c79c35ba58052cacab6709e8682a60f1 columnFamilyName B 2023-03-20 10:56:22,214 INFO [StoreOpener-c79c35ba58052cacab6709e8682a60f1-1] regionserver.HStore(310): Store=c79c35ba58052cacab6709e8682a60f1/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-20 10:56:22,214 INFO [StoreOpener-c79c35ba58052cacab6709e8682a60f1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region c79c35ba58052cacab6709e8682a60f1 2023-03-20 10:56:22,216 DEBUG [StoreOpener-c79c35ba58052cacab6709e8682a60f1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C 2023-03-20 10:56:22,216 DEBUG [StoreOpener-c79c35ba58052cacab6709e8682a60f1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C 2023-03-20 10:56:22,216 INFO [StoreOpener-c79c35ba58052cacab6709e8682a60f1-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-20 10:56:22,217 INFO [StoreOpener-c79c35ba58052cacab6709e8682a60f1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c79c35ba58052cacab6709e8682a60f1 columnFamilyName C 2023-03-20 10:56:22,218 INFO [StoreOpener-c79c35ba58052cacab6709e8682a60f1-1] regionserver.HStore(310): Store=c79c35ba58052cacab6709e8682a60f1/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-20 10:56:22,218 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:22,220 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1 2023-03-20 10:56:22,221 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1 2023-03-20 10:56:22,224 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-20 10:56:22,226 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for c79c35ba58052cacab6709e8682a60f1 2023-03-20 10:56:22,230 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-20 10:56:22,231 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened c79c35ba58052cacab6709e8682a60f1; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65928486, jitterRate=-0.01758900284767151}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-20 10:56:22,231 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for c79c35ba58052cacab6709e8682a60f1: 2023-03-20 10:56:22,233 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1., pid=11, masterSystemTime=1679309782194 2023-03-20 10:56:22,235 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:22,235 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:22,236 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=c79c35ba58052cacab6709e8682a60f1, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:22,236 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679309782236"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1679309782236"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679309782236"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679309782236"}]},"ts":"1679309782236"} 2023-03-20 10:56:22,243 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-03-20 10:56:22,244 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 in 199 msec 2023-03-20 10:56:22,249 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-03-20 10:56:22,250 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=c79c35ba58052cacab6709e8682a60f1, ASSIGN in 363 msec 2023-03-20 10:56:22,251 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-20 10:56:22,251 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679309782251"}]},"ts":"1679309782251"} 2023-03-20 10:56:22,254 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-20 10:56:22,258 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-20 10:56:22,261 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 849 msec 2023-03-20 10:56:22,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-20 10:56:22,545 INFO [Listener at localhost.localdomain/43509] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 9 completed 2023-03-20 10:56:22,550 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(139): Connect 0x18660fc4 to 127.0.0.1:60226 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 10:56:22,554 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7b6bffe4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 10:56:22,557 DEBUG [Listener at localhost.localdomain/43509] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 10:56:22,562 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41366, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 10:56:22,565 DEBUG [Listener at localhost.localdomain/43509] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-20 10:56:22,568 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:47924, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-20 10:56:22,578 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(139): Connect 0x05be6782 to 127.0.0.1:60226 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 10:56:22,583 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@449cbd3d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 10:56:22,583 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(139): Connect 0x20e4e4f3 to 127.0.0.1:60226 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 10:56:22,594 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@77443fb0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 10:56:22,595 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(139): Connect 0x13e837e0 to 127.0.0.1:60226 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 10:56:22,603 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@45864842, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 10:56:22,603 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(139): Connect 0x6362e5a0 to 127.0.0.1:60226 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 10:56:22,611 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4654132d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 10:56:22,612 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(139): Connect 0x6be8c20b to 127.0.0.1:60226 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 10:56:22,620 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@66ca212c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 10:56:22,621 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(139): Connect 0x127a87ef to 127.0.0.1:60226 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 10:56:22,629 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4cbd52be, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 10:56:22,629 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(139): Connect 0x5fe050a3 to 127.0.0.1:60226 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 10:56:22,701 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@19e3f14f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 10:56:22,703 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(139): Connect 0x38ebb9f9 to 127.0.0.1:60226 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 10:56:22,707 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3e044e22, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 10:56:22,708 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(139): Connect 0x13112432 to 127.0.0.1:60226 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 10:56:22,712 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@76d073d8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 10:56:22,717 DEBUG [hconnection-0x1023f0e6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 10:56:22,723 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41376, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 10:56:22,725 DEBUG [hconnection-0x1f7ac789-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 10:56:22,726 DEBUG [hconnection-0x70e0d6fe-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 10:56:22,732 DEBUG [hconnection-0x1c7d3439-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 10:56:22,734 DEBUG [hconnection-0x3bd05274-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 10:56:22,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:22,737 DEBUG [hconnection-0x3bed3955-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 10:56:22,737 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41388, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 10:56:22,739 DEBUG [hconnection-0x576d8a63-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 10:56:22,740 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41392, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 10:56:22,748 DEBUG [hconnection-0x4137450a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 10:56:22,748 DEBUG [hconnection-0x50dd9c42-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 10:56:22,748 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41408, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 10:56:22,757 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41416, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 10:56:22,760 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41442, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 10:56:22,761 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41428, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 10:56:22,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:22,765 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:22,765 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:22,767 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:22,768 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:22,769 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41436, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 10:56:22,771 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:22,773 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:22,773 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:41456, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 10:56:22,775 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:22,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:22,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:22,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:22,777 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:22,778 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:22,778 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:22,779 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:22,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:22,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:22,780 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:22,790 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:22,794 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:22,794 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:22,802 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:22,807 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:22,807 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:22,811 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:22,811 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. started... 2023-03-20 10:56:22,813 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c79c35ba58052cacab6709e8682a60f1 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-03-20 10:56:22,828 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=A 2023-03-20 10:56:22,829 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:22,829 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=B 2023-03-20 10:56:22,829 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:22,829 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=C 2023-03-20 10:56:22,829 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:22,841 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. as already flushing 2023-03-20 10:56:22,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on c79c35ba58052cacab6709e8682a60f1 2023-03-20 10:56:22,972 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679309782906, max timestamp=1679309782953Num uniques 859; 2023-03-20 10:56:22,972 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-20 10:56:22,972 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679309782906, max timestamp=1679309782953Num uniques 859; 2023-03-20 10:56:22,973 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-20 10:56:22,977 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679309782906, max timestamp=1679309782953Num uniques 859; 2023-03-20 10:56:22,977 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-20 10:56:23,069 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:23,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41442 deadline: 1679309843054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:23,071 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=182, cellSize=8332, totalHeapSize=19168, min timestamp=1679309783007, max timestamp=1679309783052Num uniques 1068; 2023-03-20 10:56:23,071 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=8332, suffix heap size=139040,new segment heap size=19168 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=182 2023-03-20 10:56:23,072 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679309783007, max timestamp=1679309783052Num uniques 1018; 2023-03-20 10:56:23,072 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-20 10:56:23,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679309783007, max timestamp=1679309783052Num uniques 1068; 2023-03-20 10:56:23,077 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-03-20 10:56:23,163 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:23,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41436 deadline: 1679309843154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:23,164 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679309783089, max timestamp=1679309783153Num uniques 1027; 2023-03-20 10:56:23,164 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-03-20 10:56:23,165 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679309783089, max timestamp=1679309783154Num uniques 1027; 2023-03-20 10:56:23,165 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-03-20 10:56:23,167 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679309783089, max timestamp=1679309783154Num uniques 1041; 2023-03-20 10:56:23,167 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=47670, new segment data size=6870, suffix heap size=136232,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1041, new segment cells count=150 2023-03-20 10:56:23,243 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679309783202, max timestamp=1679309783227Num uniques 1036; 2023-03-20 10:56:23,243 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=6870, suffix heap size=135712,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=150 2023-03-20 10:56:23,245 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=157, cellSize=7185, totalHeapSize=16568, min timestamp=1679309783202, max timestamp=1679309783228Num uniques 1009; 2023-03-20 10:56:23,245 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=7185, suffix heap size=132904,new segment heap size=16568 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=157 2023-03-20 10:56:23,245 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1679309783202, max timestamp=1679309783228Num uniques 1036; 2023-03-20 10:56:23,246 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-03-20 10:56:23,291 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1679309783274, max timestamp=1679309783287Num uniques 1045; 2023-03-20 10:56:23,292 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-03-20 10:56:23,292 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1679309783274, max timestamp=1679309783287Num uniques 1009; 2023-03-20 10:56:23,292 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1679309783274, max timestamp=1679309783287Num uniques 1016; 2023-03-20 10:56:23,292 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=46536, new segment data size=8930, suffix heap size=133632,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1016, new segment cells count=195 2023-03-20 10:56:23,292 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8930, suffix heap size=132904,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=195 2023-03-20 10:56:23,345 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=9 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/A/6f8496f473744f93b6ee28aef555c04f 2023-03-20 10:56:23,356 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679309783331, max timestamp=1679309783353Num uniques 1054; 2023-03-20 10:56:23,356 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-03-20 10:56:23,356 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679309783331, max timestamp=1679309783352Num uniques 1054; 2023-03-20 10:56:23,356 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679309783331, max timestamp=1679309783353Num uniques 1054; 2023-03-20 10:56:23,356 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-03-20 10:56:23,356 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-03-20 10:56:23,400 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1679309783382, max timestamp=1679309783395Num uniques 1013; 2023-03-20 10:56:23,400 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-03-20 10:56:23,402 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1679309783382, max timestamp=1679309783395Num uniques 1013; 2023-03-20 10:56:23,402 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-03-20 10:56:23,408 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1679309783382, max timestamp=1679309783395Num uniques 1013; 2023-03-20 10:56:23,408 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-03-20 10:56:23,477 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:23,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41442 deadline: 1679309843459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:23,477 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1679309783433, max timestamp=1679309783461Num uniques 1072; 2023-03-20 10:56:23,483 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=7872, suffix heap size=139456,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=172 2023-03-20 10:56:23,477 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1679309783433, max timestamp=1679309783461Num uniques 1072; 2023-03-20 10:56:23,483 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=7872, suffix heap size=139456,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=172 2023-03-20 10:56:23,486 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679309783433, max timestamp=1679309783461Num uniques 1072; 2023-03-20 10:56:23,487 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=6870, suffix heap size=139456,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=150 2023-03-20 10:56:23,504 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=9 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/B/f9d07964968a4977a1c53e7fc59cb66b 2023-03-20 10:56:23,536 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1679309783499, max timestamp=1679309783522Num uniques 1031; 2023-03-20 10:56:23,537 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1679309783499, max timestamp=1679309783522Num uniques 1031; 2023-03-20 10:56:23,537 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-03-20 10:56:23,537 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-03-20 10:56:23,538 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1679309783499, max timestamp=1679309783528Num uniques 1009; 2023-03-20 10:56:23,539 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8286, suffix heap size=132904,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=181 2023-03-20 10:56:23,590 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=9 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/C/1ace291dac7c4e52ac81c456d1c2ef03 2023-03-20 10:56:23,621 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/A/6f8496f473744f93b6ee28aef555c04f as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/6f8496f473744f93b6ee28aef555c04f 2023-03-20 10:56:23,629 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=340, cellSize=15570, totalHeapSize=35600, min timestamp=1679309783575, max timestamp=1679309783625Num uniques 1040; 2023-03-20 10:56:23,629 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=15570, suffix heap size=136128,new segment heap size=35600 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=340 2023-03-20 10:56:23,632 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=340, cellSize=15570, totalHeapSize=35600, min timestamp=1679309783575, max timestamp=1679309783625Num uniques 1040; 2023-03-20 10:56:23,632 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=15570, suffix heap size=136128,new segment heap size=35600 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=340 2023-03-20 10:56:23,636 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=340, cellSize=15570, totalHeapSize=35600, min timestamp=1679309783575, max timestamp=1679309783625Num uniques 1040; 2023-03-20 10:56:23,639 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=15570, suffix heap size=136128,new segment heap size=35600 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=340 2023-03-20 10:56:23,646 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/6f8496f473744f93b6ee28aef555c04f, entries=150, sequenceid=9, filesize=11.6 K 2023-03-20 10:56:23,651 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/B/f9d07964968a4977a1c53e7fc59cb66b as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/f9d07964968a4977a1c53e7fc59cb66b 2023-03-20 10:56:23,681 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/f9d07964968a4977a1c53e7fc59cb66b, entries=150, sequenceid=9, filesize=11.6 K 2023-03-20 10:56:23,698 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/C/1ace291dac7c4e52ac81c456d1c2ef03 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/1ace291dac7c4e52ac81c456d1c2ef03 2023-03-20 10:56:23,713 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/1ace291dac7c4e52ac81c456d1c2ef03, entries=150, sequenceid=9, filesize=11.6 K 2023-03-20 10:56:23,736 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=93.93 KB/96180 for c79c35ba58052cacab6709e8682a60f1 in 923ms, sequenceid=9, compaction requested=false 2023-03-20 10:56:23,738 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-03-20 10:56:23,741 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c79c35ba58052cacab6709e8682a60f1: 2023-03-20 10:56:23,741 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:23,742 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:23,750 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:23,750 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:23,750 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:23,750 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:23,757 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:23,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:23,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:23,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:23,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:23,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:23,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:23,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:23,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:23,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:23,761 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:23,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68410642[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:23,761 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:23,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on c79c35ba58052cacab6709e8682a60f1 2023-03-20 10:56:23,764 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c79c35ba58052cacab6709e8682a60f1 3/3 column families, dataSize=100.63 KB heapSize=268.41 KB 2023-03-20 10:56:23,764 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=A 2023-03-20 10:56:23,765 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:56:23,765 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=B 2023-03-20 10:56:23,765 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:56:23,765 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=C 2023-03-20 10:56:23,765 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:56:23,767 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:23,769 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:23,769 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:23,769 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:23,769 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:23,769 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:23,769 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:23,769 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:23,769 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:23,778 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:23,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:23,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:23,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:23,782 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:23,782 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:23,782 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:23,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:23,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:23,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:23,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:23,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:23,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:23,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:23,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:23,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:23,839 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:23,839 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:23,839 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:23,839 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:23,839 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:23,860 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:23,860 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:23,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:23,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:23,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:23,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:23,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:23,861 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:23,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:23,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:23,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:23,872 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:23,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:23,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:23,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:23,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:23,877 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:23,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:23,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:23,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:23,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:23,888 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:23,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41456 deadline: 1679309843859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:23,896 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:23,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41436 deadline: 1679309843873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:23,897 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:23,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41416 deadline: 1679309843875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:23,898 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:23,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41388 deadline: 1679309843878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:23,899 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:23,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41442 deadline: 1679309843882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:23,913 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:23,913 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:23,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:23,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:23,913 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:23,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:23,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:23,917 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:23,917 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:23,917 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:23,917 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:23,917 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:23,917 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:23,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:23,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:23,918 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:23,918 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:23,921 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:23,921 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:24,004 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:24,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41456 deadline: 1679309843995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,011 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:24,012 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:24,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41416 deadline: 1679309844001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41436 deadline: 1679309844003, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,012 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:24,013 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:24,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41442 deadline: 1679309844003, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41388 deadline: 1679309844004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,021 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:24,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:24,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:24,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:24,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:24,078 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:24,078 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:24,079 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:24,079 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:24,080 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:24,080 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:24,082 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:24,082 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:24,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:24,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:24,083 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,083 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:24,083 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:24,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:24,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:24,084 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:24,084 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:24,084 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:24,084 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:24,084 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:24,087 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:24,099 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:24,099 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:24,099 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. started... 2023-03-20 10:56:24,099 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:24,099 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. as already flushing 2023-03-20 10:56:24,100 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:24,118 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:24,118 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:24,118 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,118 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:24,118 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:24,123 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:24,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:24,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:24,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:24,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:24,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:24,126 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:24,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:24,127 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28d728c0[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:24,127 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:24,127 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:24,136 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:24,136 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:24,136 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:24,136 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:24,136 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:24,136 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:24,136 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,136 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:24,143 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,148 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:24,149 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:24,149 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:24,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:24,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:24,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:24,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:24,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:24,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:24,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:24,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:24,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:24,162 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:24,162 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:24,162 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:24,162 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:24,162 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:24,165 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:24,165 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:24,165 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:24,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:24,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:24,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:24,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:24,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:24,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:24,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:24,166 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:24,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:24,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:24,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:24,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:24,169 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:24,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:24,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,178 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,178 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:24,178 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:24,178 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,178 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:24,178 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:24,178 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:24,178 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:24,178 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:24,179 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:24,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:24,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:24,179 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:56:24,179 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:24,179 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:24,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:24,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:24,181 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:24,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:24,181 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:24,211 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:24,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41456 deadline: 1679309844208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,219 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:24,220 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:24,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41416 deadline: 1679309844216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41436 deadline: 1679309844216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,221 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:24,221 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:24,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41388 deadline: 1679309844217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41442 deadline: 1679309844217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,268 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=192 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/A/52797d30340d48f28cf1b73545a98c45 2023-03-20 10:56:24,283 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:24,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:24,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:24,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:24,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:24,337 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:24,337 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:24,338 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:24,338 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:24,339 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:24,339 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:24,340 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,340 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:24,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:24,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:24,341 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,341 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:24,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:24,342 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:24,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:24,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:24,342 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:24,361 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:24,361 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:24,361 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:24,361 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:24,375 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:24,375 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:24,375 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:24,375 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:24,375 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. started... 2023-03-20 10:56:24,375 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. as already flushing 2023-03-20 10:56:24,375 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:24,377 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:24,377 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:24,377 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,377 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:24,377 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:24,390 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:24,393 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:24,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:24,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:24,395 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:24,395 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:24,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:24,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:24,396 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=192 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/B/44209885eda1465390efd062337c353e 2023-03-20 10:56:24,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:24,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:24,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e1e5746[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:24,398 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:24,398 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:24,402 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:24,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:24,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:24,403 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:24,403 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:24,403 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:24,403 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,403 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:24,406 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:24,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:24,406 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:24,406 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:24,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:24,407 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:24,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:24,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:24,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:24,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:24,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:24,416 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:24,416 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:24,416 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:24,416 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:24,416 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:24,416 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:24,419 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:24,419 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:24,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:24,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:24,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:24,419 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:24,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:24,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:24,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:24,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:24,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:24,420 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:24,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:24,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:24,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:24,422 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:24,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:24,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,443 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,443 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:24,443 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:24,444 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:56:24,443 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:24,443 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:24,445 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:24,445 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,445 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:24,445 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:24,445 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:24,445 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:24,445 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:24,445 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:24,445 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:24,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:24,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:24,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:24,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:24,466 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=192 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/C/1f5f897a388c4cc4851260140369eb2f 2023-03-20 10:56:24,477 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/A/52797d30340d48f28cf1b73545a98c45 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/52797d30340d48f28cf1b73545a98c45 2023-03-20 10:56:24,489 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/52797d30340d48f28cf1b73545a98c45, entries=350, sequenceid=192, filesize=21.1 K 2023-03-20 10:56:24,494 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/B/44209885eda1465390efd062337c353e as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/44209885eda1465390efd062337c353e 2023-03-20 10:56:24,510 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/44209885eda1465390efd062337c353e, entries=150, sequenceid=192, filesize=11.8 K 2023-03-20 10:56:24,512 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/C/1f5f897a388c4cc4851260140369eb2f as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/1f5f897a388c4cc4851260140369eb2f 2023-03-20 10:56:24,525 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/1f5f897a388c4cc4851260140369eb2f, entries=150, sequenceid=192, filesize=11.8 K 2023-03-20 10:56:24,526 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:24,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41456 deadline: 1679309844518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,527 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~268.36 KB/274800, currentSize=87.22 KB/89310 for c79c35ba58052cacab6709e8682a60f1 in 763ms, sequenceid=192, compaction requested=false 2023-03-20 10:56:24,528 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c79c35ba58052cacab6709e8682a60f1: 2023-03-20 10:56:24,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on c79c35ba58052cacab6709e8682a60f1 2023-03-20 10:56:24,538 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c79c35ba58052cacab6709e8682a60f1 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-20 10:56:24,538 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=A 2023-03-20 10:56:24,538 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:24,538 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=B 2023-03-20 10:56:24,538 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:24,538 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=C 2023-03-20 10:56:24,538 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:24,545 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:24,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:24,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:24,579 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:24,580 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:24,579 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:24,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41442 deadline: 1679309844573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41388 deadline: 1679309844573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,592 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:24,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41416 deadline: 1679309844579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:24,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:24,598 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:24,598 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:24,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41436 deadline: 1679309844571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,606 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:24,622 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:24,623 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:24,623 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:24,635 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,636 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:24,636 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,636 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:24,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:24,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:24,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:24,638 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:24,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:24,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:24,638 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:24,639 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:24,639 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:24,639 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:24,639 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:24,644 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:24,644 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:24,644 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:24,650 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:24,651 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. started... 2023-03-20 10:56:24,658 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. as already flushing 2023-03-20 10:56:24,658 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:24,661 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:24,663 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:24,663 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,663 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:24,663 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:24,666 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:24,666 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:24,666 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:24,668 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:24,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:24,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:24,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:24,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:24,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:24,671 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:24,671 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f2f5f97[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:24,671 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:24,671 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:24,672 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:24,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:24,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:24,672 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:24,672 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:24,672 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:24,679 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:24,679 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:24,679 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:24,680 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,680 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:24,680 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:24,680 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:24,680 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:24,680 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:24,680 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:24,680 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:24,686 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:24,686 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:24,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:24,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:24,687 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:24,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:24,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:24,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:24,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:24,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:24,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:24,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:24,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:24,689 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:24,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:24,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:24,693 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:24,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,695 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:24,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41442 deadline: 1679309844695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,696 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:24,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41416 deadline: 1679309844695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,700 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:24,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41388 deadline: 1679309844700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,707 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:24,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41436 deadline: 1679309844707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,716 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:24,716 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:24,716 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:24,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:24,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:24,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:24,716 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:24,717 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:24,717 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,717 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:24,717 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:24,717 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:24,716 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:24,717 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:24,717 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:24,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:24,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:24,829 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:24,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:24,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:24,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:24,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:24,882 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:24,882 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:24,885 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:24,885 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:24,886 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:24,886 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:24,888 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,888 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:24,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:24,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:24,889 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,889 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:24,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:24,889 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:24,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:24,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:24,890 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:24,896 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:24,896 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:24,896 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:24,896 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:24,897 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:24,902 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:24,902 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:24,902 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:24,902 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:24,902 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. started... 2023-03-20 10:56:24,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41416 deadline: 1679309844900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,902 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. as already flushing 2023-03-20 10:56:24,909 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:24,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41388 deadline: 1679309844904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,910 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:24,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41442 deadline: 1679309844906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,912 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:24,912 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:24,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41436 deadline: 1679309844912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,923 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:24,923 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:24,923 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,923 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:24,923 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:24,925 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,926 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:24,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:24,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:24,926 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:24,927 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:24,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:24,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:24,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:24,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:24,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:24,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f3c3726[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:24,930 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:24,930 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:24,944 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:24,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:24,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:24,944 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:24,944 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:24,944 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:24,944 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,944 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:24,946 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,946 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:24,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:24,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:24,946 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:24,946 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:24,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:24,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:24,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:24,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:24,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:24,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:24,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:24,959 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:24,959 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:24,959 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:24,959 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:24,959 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:24,969 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:24,969 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:24,969 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:24,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:24,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:24,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:24,970 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:24,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:24,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:24,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:24,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:24,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:24,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:24,971 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:24,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:24,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:24,978 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:24,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:24,978 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:24,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:24,979 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,979 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:24,979 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:24,979 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:24,979 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:24,979 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:24,979 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:24,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:24,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:24,980 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:24,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:24,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:24,982 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:24,982 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:56:24,982 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:25,010 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-03-20 10:56:25,011 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-03-20 10:56:25,030 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:25,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41456 deadline: 1679309845028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:25,056 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=209 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/A/780061fdcd3c4cdebd25ad2b6d906b02 2023-03-20 10:56:25,083 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:25,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:25,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:25,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:25,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:25,116 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:25,116 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:25,117 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:25,117 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:25,119 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:25,119 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:25,120 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:25,120 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:25,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:25,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:25,120 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:25,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:25,120 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:25,121 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:25,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:25,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:25,121 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:25,122 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:25,122 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:25,122 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:25,122 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:25,123 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:25,124 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:25,124 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:25,124 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:25,124 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. started... 2023-03-20 10:56:25,126 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. as already flushing 2023-03-20 10:56:25,126 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:25,130 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:25,130 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:25,130 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:25,131 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:25,131 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:25,135 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:25,135 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:25,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:25,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:25,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:25,136 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:25,136 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:25,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:25,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:25,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:25,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:25,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:25,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:25,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e3c8f29[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:25,139 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:25,139 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:25,144 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:25,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:25,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:25,144 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:25,144 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:25,144 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:25,144 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:25,145 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:25,152 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:25,152 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:25,152 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:25,152 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:25,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:25,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:25,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:25,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:25,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:25,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:25,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:25,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:25,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:25,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:25,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:25,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:25,157 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:25,157 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:25,157 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:25,157 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:25,157 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:25,161 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:25,161 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:25,161 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:25,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:25,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:25,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:25,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:25,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:25,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:25,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:25,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:25,163 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:25,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:25,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:25,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:25,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:25,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:25,165 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:25,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:25,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:25,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:25,171 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:25,171 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:25,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:25,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:25,171 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:25,171 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:25,171 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:25,171 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:25,171 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:25,171 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:25,171 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:25,172 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:25,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:25,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:25,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:25,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:25,179 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:25,179 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:25,209 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:25,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41416 deadline: 1679309845208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:25,214 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:25,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41388 deadline: 1679309845212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:25,214 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:25,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41442 deadline: 1679309845213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:25,218 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:25,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41436 deadline: 1679309845216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:25,279 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:25,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:25,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:25,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:25,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:25,312 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:25,312 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:25,314 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:25,314 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:25,315 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:25,315 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:25,316 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:25,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:25,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:25,317 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:25,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:25,318 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:25,318 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:25,318 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:25,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:25,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:25,318 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:25,319 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:25,319 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:25,319 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:25,319 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:25,320 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:25,326 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:25,326 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:25,336 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:25,336 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. started... 2023-03-20 10:56:25,336 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. as already flushing 2023-03-20 10:56:25,336 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:25,348 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:25,348 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:25,348 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:25,348 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:25,348 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:25,355 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:25,355 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:25,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:25,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:25,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:25,356 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:25,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:25,356 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:25,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:25,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:25,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:25,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:25,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:25,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62645fcf[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:25,360 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:25,360 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:25,372 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:25,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:25,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:25,375 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:25,375 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:25,375 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:25,376 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:25,376 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:25,378 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:25,378 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:25,378 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:25,378 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:25,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:25,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:25,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:25,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:25,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:25,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:25,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:25,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:25,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:25,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:25,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:25,409 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:25,409 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:25,410 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:25,410 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:25,410 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:25,410 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:25,411 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:25,411 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:25,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:25,411 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:25,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:25,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:25,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:25,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:25,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:25,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:25,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:25,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:25,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:25,414 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:25,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:25,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:25,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:25,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:25,415 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:25,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:25,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:25,419 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:25,419 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:25,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:25,419 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:25,419 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:25,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:25,421 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:25,419 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:25,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:25,421 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:25,421 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:25,421 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:25,421 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:25,421 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:25,421 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:25,422 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:25,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:25,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:25,516 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=209 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/B/eb6f0d88534d474f87a0af1e88c0a639 2023-03-20 10:56:25,526 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:25,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:25,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:25,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:25,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:25,545 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:25,545 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:25,546 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:25,546 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:25,550 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:25,550 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:25,554 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:25,554 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:25,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:25,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:25,554 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:25,554 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:25,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:25,555 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:25,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:25,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:25,556 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:25,558 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:25,558 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:25,558 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:25,558 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:25,563 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:25,566 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:25,566 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:25,582 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:25,583 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. started... 2023-03-20 10:56:25,583 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. as already flushing 2023-03-20 10:56:25,583 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:25,586 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:25,586 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:25,586 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:25,586 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:25,586 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:25,594 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:25,594 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:25,595 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:25,595 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:25,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:25,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:25,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:25,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:25,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:25,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:25,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:25,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:25,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:25,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77a267f3[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:25,614 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:25,614 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:25,616 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:25,616 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:25,617 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:25,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:25,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:25,617 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:25,617 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:25,617 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:25,634 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:25,635 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:25,635 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:25,636 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:25,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:25,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:25,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:25,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:25,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:25,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:25,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:25,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:25,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:25,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:25,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:25,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:25,649 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:25,649 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:25,649 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:25,649 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:25,649 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:25,651 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:25,651 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:25,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:25,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:25,651 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:25,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:25,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:25,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:25,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:25,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:25,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:25,653 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:25,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:25,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:25,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:25,660 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:25,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:25,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:25,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:25,676 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=209 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/C/e8cf37fb88d148c1b0d8095add427b70 2023-03-20 10:56:25,687 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:25,687 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:25,687 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:25,687 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:25,687 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:25,700 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:25,700 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:25,700 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:25,699 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:25,700 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:25,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:25,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:25,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:25,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:25,701 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:25,701 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:25,708 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/A/780061fdcd3c4cdebd25ad2b6d906b02 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/780061fdcd3c4cdebd25ad2b6d906b02 2023-03-20 10:56:25,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:25,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:25,710 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:56:25,720 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:25,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41416 deadline: 1679309845718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:25,721 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:25,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41388 deadline: 1679309845720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:25,733 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/780061fdcd3c4cdebd25ad2b6d906b02, entries=200, sequenceid=209, filesize=14.1 K 2023-03-20 10:56:25,735 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/B/eb6f0d88534d474f87a0af1e88c0a639 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/eb6f0d88534d474f87a0af1e88c0a639 2023-03-20 10:56:25,736 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:25,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41442 deadline: 1679309845723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:25,737 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:25,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41436 deadline: 1679309845721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:25,762 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/eb6f0d88534d474f87a0af1e88c0a639, entries=150, sequenceid=209, filesize=11.8 K 2023-03-20 10:56:25,766 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/C/e8cf37fb88d148c1b0d8095add427b70 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/e8cf37fb88d148c1b0d8095add427b70 2023-03-20 10:56:25,801 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:25,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:25,806 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/e8cf37fb88d148c1b0d8095add427b70, entries=150, sequenceid=209, filesize=11.8 K 2023-03-20 10:56:25,808 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for c79c35ba58052cacab6709e8682a60f1 in 1270ms, sequenceid=209, compaction requested=true 2023-03-20 10:56:25,808 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c79c35ba58052cacab6709e8682a60f1: 2023-03-20 10:56:25,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:25,826 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:56:25,826 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:56:25,826 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:56:25,827 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 10:56:25,829 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 10:56:25,836 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35958 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 10:56:25,839 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): c79c35ba58052cacab6709e8682a60f1/B is initiating minor compaction (all files) 2023-03-20 10:56:25,839 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c79c35ba58052cacab6709e8682a60f1/B in TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:25,839 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/f9d07964968a4977a1c53e7fc59cb66b, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/44209885eda1465390efd062337c353e, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/eb6f0d88534d474f87a0af1e88c0a639] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp, totalSize=35.1 K 2023-03-20 10:56:25,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:25,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:25,852 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:25,857 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:25,854 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting f9d07964968a4977a1c53e7fc59cb66b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=9, earliestPutTs=1679309782788 2023-03-20 10:56:25,859 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:25,866 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:25,867 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:25,867 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:25,869 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:25,869 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:25,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:25,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:25,869 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:25,869 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:25,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:25,870 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:25,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:25,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:25,870 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:25,881 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:25,881 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:25,882 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:25,882 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:25,888 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:25,892 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:25,892 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:25,892 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:25,892 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. started... 2023-03-20 10:56:25,892 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c79c35ba58052cacab6709e8682a60f1 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-20 10:56:25,893 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=A 2023-03-20 10:56:25,893 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:25,893 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=B 2023-03-20 10:56:25,893 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:25,893 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=C 2023-03-20 10:56:25,893 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:25,893 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 47912 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 10:56:25,894 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1912): c79c35ba58052cacab6709e8682a60f1/A is initiating minor compaction (all files) 2023-03-20 10:56:25,894 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c79c35ba58052cacab6709e8682a60f1/A in TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:25,894 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/6f8496f473744f93b6ee28aef555c04f, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/52797d30340d48f28cf1b73545a98c45, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/780061fdcd3c4cdebd25ad2b6d906b02] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp, totalSize=46.8 K 2023-03-20 10:56:25,895 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 44209885eda1465390efd062337c353e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=192, earliestPutTs=1679309783643 2023-03-20 10:56:25,896 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 6f8496f473744f93b6ee28aef555c04f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=9, earliestPutTs=1679309782788 2023-03-20 10:56:25,905 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 52797d30340d48f28cf1b73545a98c45, keycount=350, bloomtype=ROW, size=21.1 K, encoding=NONE, compression=NONE, seqNum=192, earliestPutTs=1679309783617 2023-03-20 10:56:25,905 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-20 10:56:25,906 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 780061fdcd3c4cdebd25ad2b6d906b02, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1679309783829 2023-03-20 10:56:25,914 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting eb6f0d88534d474f87a0af1e88c0a639, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1679309783829 2023-03-20 10:56:26,021 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] throttle.PressureAwareThroughputController(145): c79c35ba58052cacab6709e8682a60f1#A#compaction#10 average throughput is 0.66 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:56:26,080 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): c79c35ba58052cacab6709e8682a60f1#B#compaction#11 average throughput is 0.10 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:56:26,089 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. as already flushing 2023-03-20 10:56:26,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on c79c35ba58052cacab6709e8682a60f1 2023-03-20 10:56:26,109 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=224 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/A/0744e7335f794d89b78d0b8374b959ae 2023-03-20 10:56:26,239 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-03-20 10:56:26,239 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-03-20 10:56:26,373 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/B/ff4926af8be244959805131528248237 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/ff4926af8be244959805131528248237 2023-03-20 10:56:26,382 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:26,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41456 deadline: 1679309846377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:26,415 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c79c35ba58052cacab6709e8682a60f1/B of c79c35ba58052cacab6709e8682a60f1 into ff4926af8be244959805131528248237(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:56:26,422 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c79c35ba58052cacab6709e8682a60f1: 2023-03-20 10:56:26,423 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1., storeName=c79c35ba58052cacab6709e8682a60f1/B, priority=13, startTime=1679309785826; duration=0sec 2023-03-20 10:56:26,423 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:56:26,424 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 10:56:26,431 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35958 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 10:56:26,431 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): c79c35ba58052cacab6709e8682a60f1/C is initiating minor compaction (all files) 2023-03-20 10:56:26,431 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c79c35ba58052cacab6709e8682a60f1/C in TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:26,432 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/1ace291dac7c4e52ac81c456d1c2ef03, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/1f5f897a388c4cc4851260140369eb2f, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/e8cf37fb88d148c1b0d8095add427b70] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp, totalSize=35.1 K 2023-03-20 10:56:26,433 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 1ace291dac7c4e52ac81c456d1c2ef03, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=9, earliestPutTs=1679309782788 2023-03-20 10:56:26,434 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 1f5f897a388c4cc4851260140369eb2f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=192, earliestPutTs=1679309783643 2023-03-20 10:56:26,437 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting e8cf37fb88d148c1b0d8095add427b70, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1679309783829 2023-03-20 10:56:26,476 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): c79c35ba58052cacab6709e8682a60f1#C#compaction#13 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:56:26,488 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:26,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41456 deadline: 1679309846488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:26,540 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/C/863198e70a3d4dca9ec1055e36d1dcd2 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/863198e70a3d4dca9ec1055e36d1dcd2 2023-03-20 10:56:26,554 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c79c35ba58052cacab6709e8682a60f1/C of c79c35ba58052cacab6709e8682a60f1 into 863198e70a3d4dca9ec1055e36d1dcd2(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:56:26,554 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c79c35ba58052cacab6709e8682a60f1: 2023-03-20 10:56:26,554 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1., storeName=c79c35ba58052cacab6709e8682a60f1/C, priority=13, startTime=1679309785826; duration=0sec 2023-03-20 10:56:26,554 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:56:26,612 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/A/ed58a95cba484f9a8b766ded56c5b6f0 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/ed58a95cba484f9a8b766ded56c5b6f0 2023-03-20 10:56:26,633 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c79c35ba58052cacab6709e8682a60f1/A of c79c35ba58052cacab6709e8682a60f1 into ed58a95cba484f9a8b766ded56c5b6f0(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:56:26,633 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c79c35ba58052cacab6709e8682a60f1: 2023-03-20 10:56:26,633 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1., storeName=c79c35ba58052cacab6709e8682a60f1/A, priority=13, startTime=1679309785810; duration=0sec 2023-03-20 10:56:26,633 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:56:26,699 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:26,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41456 deadline: 1679309846696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:26,750 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:26,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41416 deadline: 1679309846730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:26,753 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:26,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41436 deadline: 1679309846753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:26,754 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:26,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41388 deadline: 1679309846753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:26,754 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:26,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41442 deadline: 1679309846753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:26,768 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=224 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/B/6b2989c71b9d492ea297bfebf920c659 2023-03-20 10:56:26,834 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=224 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/C/58b5d8bffcd64553afcd411518ae5c63 2023-03-20 10:56:26,847 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/A/0744e7335f794d89b78d0b8374b959ae as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/0744e7335f794d89b78d0b8374b959ae 2023-03-20 10:56:26,856 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/0744e7335f794d89b78d0b8374b959ae, entries=150, sequenceid=224, filesize=11.8 K 2023-03-20 10:56:26,859 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/B/6b2989c71b9d492ea297bfebf920c659 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/6b2989c71b9d492ea297bfebf920c659 2023-03-20 10:56:26,888 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/6b2989c71b9d492ea297bfebf920c659, entries=150, sequenceid=224, filesize=11.8 K 2023-03-20 10:56:26,890 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/C/58b5d8bffcd64553afcd411518ae5c63 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/58b5d8bffcd64553afcd411518ae5c63 2023-03-20 10:56:26,921 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/58b5d8bffcd64553afcd411518ae5c63, entries=150, sequenceid=224, filesize=11.8 K 2023-03-20 10:56:26,924 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for c79c35ba58052cacab6709e8682a60f1 in 1031ms, sequenceid=224, compaction requested=false 2023-03-20 10:56:26,924 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c79c35ba58052cacab6709e8682a60f1: 2023-03-20 10:56:26,924 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:26,924 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:26,927 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:26,927 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:26,927 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:26,927 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:26,929 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:26,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:26,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:26,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:26,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:26,930 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:26,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:26,930 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:26,930 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:26,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:26,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:26,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:26,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:26,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19c9eb26[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:26,931 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:26,931 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:26,945 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:26,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:26,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:26,945 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:26,945 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:26,945 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:26,945 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:26,945 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:26,946 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:26,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:26,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:26,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:26,946 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:26,947 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:26,947 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:26,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:26,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:26,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:26,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:26,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:26,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:26,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:26,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:26,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:26,952 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:26,952 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:26,952 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:26,952 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:26,952 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:26,960 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:26,960 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:26,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:26,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:26,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:26,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:26,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:26,963 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:26,964 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:26,968 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:26,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:26,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:26,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:26,969 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:26,969 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:26,969 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:26,969 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:26,969 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:26,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:26,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:26,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:26,969 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:26,969 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:26,969 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:26,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:26,969 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:26,969 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:26,972 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:26,972 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:26,972 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:26,972 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:26,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:26,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:26,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:26,973 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:56:27,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on c79c35ba58052cacab6709e8682a60f1 2023-03-20 10:56:27,023 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c79c35ba58052cacab6709e8682a60f1 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-20 10:56:27,024 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=A 2023-03-20 10:56:27,024 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:27,024 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=B 2023-03-20 10:56:27,024 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:27,024 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=C 2023-03-20 10:56:27,024 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:27,072 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:27,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:27,079 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=245 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/A/9038b849757243ae818620cc35239708 2023-03-20 10:56:27,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:27,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:27,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:27,093 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:27,093 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:27,107 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:27,107 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:27,107 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:27,108 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:27,109 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,109 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,109 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:27,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:27,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:27,109 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:27,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:27,112 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:27,113 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:27,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41456 deadline: 1679309847110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:27,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:27,113 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:27,114 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:27,114 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:27,114 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:27,114 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:27,117 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:27,119 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:27,119 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:27,120 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. started... 2023-03-20 10:56:27,120 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:27,120 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. as already flushing 2023-03-20 10:56:27,120 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:27,120 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:27,120 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:27,120 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,120 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:27,120 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:27,122 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:27,122 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:27,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:27,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:27,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:27,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:27,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,127 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:27,127 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:27,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:27,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:27,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63f67af1[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:27,128 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:27,128 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:27,135 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:27,135 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:27,135 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:27,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:27,136 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:27,136 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:27,136 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:27,144 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:27,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:27,146 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:27,146 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:27,146 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:27,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:27,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:27,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:27,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:27,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:27,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:27,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:27,160 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:27,160 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:27,160 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:27,160 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:27,160 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:27,163 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:27,163 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:27,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:27,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:27,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:27,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:27,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:27,164 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:27,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:27,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:27,164 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:27,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:27,171 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:27,171 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=245 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/B/927f030c43e74efd99f74406ebf0ca71 2023-03-20 10:56:27,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:27,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:27,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:27,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:27,177 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,177 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:27,177 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:27,177 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,177 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:27,177 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:27,177 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:27,177 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:27,177 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:27,177 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:27,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:27,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:27,178 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:56:27,178 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:27,178 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:27,180 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:56:27,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:27,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:27,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:27,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:27,222 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:27,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41456 deadline: 1679309847216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,271 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=245 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/C/4fbc60dd9f3b4654b8f9ca1ac325a7e2 2023-03-20 10:56:27,278 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:27,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:27,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:27,288 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/A/9038b849757243ae818620cc35239708 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/9038b849757243ae818620cc35239708 2023-03-20 10:56:27,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:27,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:27,289 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:27,289 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:27,306 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:27,306 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:27,306 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:27,306 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:27,307 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,308 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:27,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:27,308 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,308 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:27,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:27,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:27,310 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:27,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:27,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:27,310 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:27,314 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:27,314 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:27,314 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:27,314 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:27,319 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:27,319 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:27,319 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:27,320 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:27,321 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. started... 2023-03-20 10:56:27,321 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. as already flushing 2023-03-20 10:56:27,321 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:27,321 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:27,321 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:27,321 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,321 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:27,321 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:27,323 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:27,323 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:27,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:27,323 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:27,323 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:27,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:27,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:27,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:27,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:27,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:27,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@232cdb67[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:27,325 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:27,325 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:27,326 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:27,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:27,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:27,326 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:27,326 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:27,326 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:27,326 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,327 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:27,328 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:27,328 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:27,328 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:27,328 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:27,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:27,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:27,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:27,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:27,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:27,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:27,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:27,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:27,331 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:27,331 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:27,331 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:27,332 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:27,332 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:27,335 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:27,336 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:27,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:27,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:27,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:27,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:27,336 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:27,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:27,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:27,336 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:27,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:27,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:27,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:27,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:27,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:27,341 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,342 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/9038b849757243ae818620cc35239708, entries=150, sequenceid=245, filesize=11.8 K 2023-03-20 10:56:27,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:27,351 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/B/927f030c43e74efd99f74406ebf0ca71 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/927f030c43e74efd99f74406ebf0ca71 2023-03-20 10:56:27,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:27,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,357 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,357 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:27,357 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:27,357 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:27,357 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:27,357 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:27,357 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:27,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:27,357 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:27,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:27,357 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,358 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:27,358 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:27,358 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:27,358 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:27,358 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:27,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:27,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:27,373 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/927f030c43e74efd99f74406ebf0ca71, entries=150, sequenceid=245, filesize=11.8 K 2023-03-20 10:56:27,379 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/C/4fbc60dd9f3b4654b8f9ca1ac325a7e2 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/4fbc60dd9f3b4654b8f9ca1ac325a7e2 2023-03-20 10:56:27,391 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/4fbc60dd9f3b4654b8f9ca1ac325a7e2, entries=150, sequenceid=245, filesize=11.8 K 2023-03-20 10:56:27,393 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for c79c35ba58052cacab6709e8682a60f1 in 370ms, sequenceid=245, compaction requested=true 2023-03-20 10:56:27,393 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c79c35ba58052cacab6709e8682a60f1: 2023-03-20 10:56:27,394 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:56:27,394 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 10:56:27,394 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 10:56:27,394 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:56:27,394 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:56:27,396 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36211 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 10:56:27,397 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1912): c79c35ba58052cacab6709e8682a60f1/A is initiating minor compaction (all files) 2023-03-20 10:56:27,397 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c79c35ba58052cacab6709e8682a60f1/A in TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:27,397 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/ed58a95cba484f9a8b766ded56c5b6f0, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/0744e7335f794d89b78d0b8374b959ae, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/9038b849757243ae818620cc35239708] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp, totalSize=35.4 K 2023-03-20 10:56:27,397 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36211 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 10:56:27,397 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): c79c35ba58052cacab6709e8682a60f1/B is initiating minor compaction (all files) 2023-03-20 10:56:27,397 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c79c35ba58052cacab6709e8682a60f1/B in TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:27,398 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/ff4926af8be244959805131528248237, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/6b2989c71b9d492ea297bfebf920c659, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/927f030c43e74efd99f74406ebf0ca71] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp, totalSize=35.4 K 2023-03-20 10:56:27,399 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting ed58a95cba484f9a8b766ded56c5b6f0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1679309783829 2023-03-20 10:56:27,399 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting ff4926af8be244959805131528248237, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1679309783829 2023-03-20 10:56:27,401 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 0744e7335f794d89b78d0b8374b959ae, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=224, earliestPutTs=1679309784562 2023-03-20 10:56:27,402 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 6b2989c71b9d492ea297bfebf920c659, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=224, earliestPutTs=1679309784562 2023-03-20 10:56:27,402 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 9038b849757243ae818620cc35239708, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1679309786370 2023-03-20 10:56:27,402 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 927f030c43e74efd99f74406ebf0ca71, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1679309786370 2023-03-20 10:56:27,416 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): c79c35ba58052cacab6709e8682a60f1#B#compaction#18 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:56:27,427 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] throttle.PressureAwareThroughputController(145): c79c35ba58052cacab6709e8682a60f1#A#compaction#19 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:56:27,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on c79c35ba58052cacab6709e8682a60f1 2023-03-20 10:56:27,429 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c79c35ba58052cacab6709e8682a60f1 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-20 10:56:27,429 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=A 2023-03-20 10:56:27,430 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:27,430 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=B 2023-03-20 10:56:27,430 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:27,430 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=C 2023-03-20 10:56:27,430 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:27,458 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:27,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:27,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:27,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:27,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:27,475 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:27,475 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:27,478 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:27,478 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:27,480 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:27,480 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:27,486 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:27,486 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:27,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:27,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:27,487 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,487 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:27,488 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:27,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:27,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:27,491 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:27,494 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:27,494 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:27,494 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:27,494 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:27,495 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:27,497 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:27,497 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:27,497 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:27,497 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. started... 2023-03-20 10:56:27,498 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. as already flushing 2023-03-20 10:56:27,498 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:27,498 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:27,498 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:27,498 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,498 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:27,498 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:27,499 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,499 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:27,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:27,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:27,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:27,500 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:27,500 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:27,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:27,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:27,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:27,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:27,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47d5ad0e[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:27,503 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:27,503 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:27,504 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:27,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:27,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:27,505 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:27,505 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:27,505 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:27,505 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,505 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:27,506 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:27,506 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:27,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:27,506 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:27,506 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:27,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:27,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:27,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:27,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:27,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:27,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:27,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:27,510 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:27,510 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:27,510 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:27,510 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:27,510 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:27,512 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:27,512 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:27,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:27,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:27,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:27,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:27,513 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:27,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:27,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:27,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:27,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:27,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:27,513 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:27,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:27,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:27,515 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:27,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:27,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,521 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:27,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41456 deadline: 1679309847520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,526 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:27,527 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:27,527 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:27,527 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:56:27,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:27,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:27,528 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:27,527 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:27,527 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:27,528 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:27,528 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,528 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:27,528 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:27,528 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:27,528 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:27,528 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:27,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:27,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:27,624 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:27,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41456 deadline: 1679309847623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,628 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:27,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:27,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:27,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:27,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:27,633 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:27,633 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:27,638 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:27,638 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:27,639 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:27,639 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:27,640 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:27,640 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:27,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:27,640 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,640 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:27,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:27,640 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:27,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:27,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:27,641 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:27,641 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:27,641 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:27,641 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:27,641 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:27,642 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:27,642 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:27,642 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:27,642 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:27,642 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. started... 2023-03-20 10:56:27,643 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. as already flushing 2023-03-20 10:56:27,643 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:27,643 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:27,643 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:27,643 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,643 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:27,643 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:27,651 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:27,651 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:27,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:27,652 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:27,652 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:27,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:27,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:27,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:27,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:27,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:27,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1193fe09[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:27,654 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:27,654 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:27,656 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:27,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:27,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:27,657 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,657 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:27,657 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:27,657 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:27,657 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:27,659 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:27,659 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,659 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:27,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:27,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:27,659 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:27,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:27,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:27,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:27,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:27,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:27,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:27,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:27,662 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:27,663 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:27,663 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:27,663 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:27,663 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:27,664 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:27,665 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:27,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:27,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:27,665 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:27,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:27,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:27,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:27,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:27,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:27,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:27,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:27,666 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:27,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:27,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:27,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:27,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:27,670 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,677 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:27,677 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:27,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:27,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:27,678 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,678 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:27,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:27,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:27,678 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:27,678 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,678 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:27,678 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:27,678 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:27,679 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:27,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:27,682 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:27,683 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:27,683 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:27,783 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:27,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:27,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:27,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:27,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:27,801 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:27,801 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:27,809 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:27,810 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:27,810 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:27,810 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:27,814 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:27,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:27,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:27,814 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:27,815 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,816 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:27,817 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:27,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:27,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:27,817 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:27,818 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:27,818 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:27,818 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:27,818 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:27,819 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:27,822 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:27,822 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:27,822 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:27,822 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. started... 2023-03-20 10:56:27,822 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. as already flushing 2023-03-20 10:56:27,822 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:27,822 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:27,822 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:27,822 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,823 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:27,823 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:27,827 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:27,827 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:27,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:27,828 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:27,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:27,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41456 deadline: 1679309847826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,828 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:27,828 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:27,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:27,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:27,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:27,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:27,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41e22f38[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:27,830 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:27,830 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:27,831 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:27,831 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:27,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:27,831 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:27,832 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:27,832 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:27,832 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:27,833 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:27,833 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,833 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:27,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:27,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:27,833 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:27,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:27,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:27,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:27,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:27,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:27,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:27,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:27,837 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:27,837 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:27,837 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:27,837 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:27,837 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:27,840 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:27,840 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:27,840 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:27,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:27,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:27,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:27,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:27,841 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:27,843 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:27,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:27,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:27,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:27,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:27,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:27,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:27,845 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:27,845 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:27,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:27,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:27,846 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,846 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:27,846 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:27,846 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,846 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:27,846 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:27,846 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:27,846 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:27,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:27,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:27,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:27,851 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:27,851 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:27,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:27,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:27,852 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:56:27,879 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/B/b02e7361f3374526b36dcdbcd3ff84d4 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/b02e7361f3374526b36dcdbcd3ff84d4 2023-03-20 10:56:27,898 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=260 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/A/ae31770d2a314d17bf700433045ca163 2023-03-20 10:56:27,903 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c79c35ba58052cacab6709e8682a60f1/B of c79c35ba58052cacab6709e8682a60f1 into b02e7361f3374526b36dcdbcd3ff84d4(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:56:27,907 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c79c35ba58052cacab6709e8682a60f1: 2023-03-20 10:56:27,907 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1., storeName=c79c35ba58052cacab6709e8682a60f1/B, priority=13, startTime=1679309787394; duration=0sec 2023-03-20 10:56:27,907 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:56:27,907 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 10:56:27,910 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36211 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 10:56:27,910 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): c79c35ba58052cacab6709e8682a60f1/C is initiating minor compaction (all files) 2023-03-20 10:56:27,910 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c79c35ba58052cacab6709e8682a60f1/C in TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:27,911 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/863198e70a3d4dca9ec1055e36d1dcd2, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/58b5d8bffcd64553afcd411518ae5c63, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/4fbc60dd9f3b4654b8f9ca1ac325a7e2] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp, totalSize=35.4 K 2023-03-20 10:56:27,914 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 863198e70a3d4dca9ec1055e36d1dcd2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1679309783829 2023-03-20 10:56:27,915 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/A/6ee6083833fc437a93190b9610c7ed05 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/6ee6083833fc437a93190b9610c7ed05 2023-03-20 10:56:27,915 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 58b5d8bffcd64553afcd411518ae5c63, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=224, earliestPutTs=1679309784562 2023-03-20 10:56:27,918 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 4fbc60dd9f3b4654b8f9ca1ac325a7e2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1679309786370 2023-03-20 10:56:27,942 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c79c35ba58052cacab6709e8682a60f1/A of c79c35ba58052cacab6709e8682a60f1 into 6ee6083833fc437a93190b9610c7ed05(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:56:27,945 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c79c35ba58052cacab6709e8682a60f1: 2023-03-20 10:56:27,946 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1., storeName=c79c35ba58052cacab6709e8682a60f1/A, priority=13, startTime=1679309787394; duration=0sec 2023-03-20 10:56:27,946 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:56:27,951 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:27,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:27,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:27,975 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): c79c35ba58052cacab6709e8682a60f1#C#compaction#22 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:56:27,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:27,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:27,977 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:27,977 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:27,978 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:27,978 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:27,979 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:27,979 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:27,983 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:27,983 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:27,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:27,984 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:27,984 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:27,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:27,989 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:27,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:27,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:27,989 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:27,990 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:27,990 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:27,990 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:27,990 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:28,001 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:28,002 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:28,002 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:28,002 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:28,002 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. started... 2023-03-20 10:56:28,003 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. as already flushing 2023-03-20 10:56:28,003 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:28,003 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:28,003 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:28,004 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,006 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:28,007 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:28,010 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:28,010 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:28,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:28,011 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:28,011 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:28,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:28,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:28,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:28,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:28,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:28,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d03380[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:28,014 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:28,014 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:28,015 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:28,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:28,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:28,016 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:28,016 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:28,016 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:28,016 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,016 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:28,018 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:28,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:28,018 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:28,018 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:28,018 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:28,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:28,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:28,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:28,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:28,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:28,024 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/C/d9e2ed4b3ac94996b6f925eccc55bcec as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/d9e2ed4b3ac94996b6f925eccc55bcec 2023-03-20 10:56:28,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:28,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:28,026 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:28,030 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:28,030 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:28,030 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:28,030 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:28,031 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:28,031 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:28,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:28,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:28,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:28,031 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:28,032 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:28,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:28,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:28,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:28,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:28,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:28,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:28,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:28,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:28,045 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:28,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:28,059 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c79c35ba58052cacab6709e8682a60f1/C of c79c35ba58052cacab6709e8682a60f1 into d9e2ed4b3ac94996b6f925eccc55bcec(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:56:28,059 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c79c35ba58052cacab6709e8682a60f1: 2023-03-20 10:56:28,059 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1., storeName=c79c35ba58052cacab6709e8682a60f1/C, priority=13, startTime=1679309787394; duration=0sec 2023-03-20 10:56:28,059 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:56:28,060 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,060 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:28,061 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:28,061 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:28,061 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,061 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:28,061 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:28,061 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:28,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:28,066 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:56:28,063 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:28,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:28,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:28,061 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:28,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:28,067 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:28,067 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:28,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:28,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:28,135 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:28,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41456 deadline: 1679309848133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,167 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:28,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:28,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:28,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:28,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:28,179 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:28,179 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:28,181 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:28,181 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:28,182 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:28,182 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:28,183 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:28,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:28,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:28,184 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:28,184 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,184 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:28,184 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:28,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:28,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:28,185 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:28,186 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:28,186 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:28,186 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:28,186 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:28,195 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:28,202 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:28,202 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:28,206 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:28,206 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. started... 2023-03-20 10:56:28,206 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. as already flushing 2023-03-20 10:56:28,207 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:28,207 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:28,207 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:28,207 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,207 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:28,207 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:28,211 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:28,212 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:28,212 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:28,218 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:28,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:28,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:28,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:28,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:28,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:28,223 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:28,223 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e0743c9[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:28,223 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:28,223 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:28,227 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:28,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:28,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:28,227 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:28,227 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:28,227 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:28,228 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,228 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:28,229 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:28,229 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:28,229 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:28,229 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:28,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:28,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:28,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:28,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:28,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:28,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:28,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:28,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:28,234 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:28,234 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:28,234 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:28,234 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:28,234 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:28,244 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:28,244 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:28,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:28,244 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:28,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:28,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:28,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:28,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:28,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:28,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:28,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:28,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:28,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:28,249 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:28,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:28,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:28,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:28,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,258 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,265 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,266 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:28,266 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:28,266 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,266 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:28,266 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:28,266 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:28,266 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:28,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:28,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:28,267 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:28,267 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:28,268 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:28,268 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:28,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:28,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:28,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:28,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:28,367 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:28,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:28,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:28,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:28,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:28,374 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:28,374 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:28,379 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:28,379 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:28,379 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:28,379 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:28,385 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:28,385 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:28,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:28,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:28,387 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,387 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:28,387 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:28,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:28,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:28,387 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:28,388 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:28,388 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:28,388 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:28,388 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:28,389 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:28,390 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:28,390 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:28,390 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:28,390 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. started... 2023-03-20 10:56:28,390 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. as already flushing 2023-03-20 10:56:28,390 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:28,390 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:28,390 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:28,391 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,391 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:28,391 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:28,392 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:28,393 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,393 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:28,394 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:28,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:28,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:28,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:28,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:28,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:28,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:28,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:28,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60ea8bb6[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:28,399 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:28,399 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:28,403 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,403 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:28,406 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:28,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:28,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:28,406 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:28,406 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:28,406 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:28,407 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=260 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/B/226c26459a444a229fa338b447e9d2b9 2023-03-20 10:56:28,412 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,412 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:28,413 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:28,413 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:28,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:28,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:28,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:28,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:28,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:28,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:28,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:28,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:28,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:28,427 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:28,427 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:28,427 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:28,427 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:28,427 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:28,430 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:28,430 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:28,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:28,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:28,430 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:28,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:28,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:28,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:28,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:28,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:28,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:28,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:28,436 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:28,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:28,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:28,438 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:28,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:28,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,444 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,444 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:28,444 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:28,444 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:56:28,444 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:28,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:28,444 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:28,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:28,444 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,444 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:28,445 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:28,445 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:28,445 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:28,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:28,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:28,446 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:28,446 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:28,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:28,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:28,535 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=260 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/C/23ba5c831bbc4eddbe9dce7adc592ee2 2023-03-20 10:56:28,549 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:28,549 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/A/ae31770d2a314d17bf700433045ca163 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/ae31770d2a314d17bf700433045ca163 2023-03-20 10:56:28,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:28,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:28,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:28,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:28,557 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:28,557 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:28,558 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:28,558 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:28,558 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:28,559 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:28,561 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:28,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:28,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:28,561 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/ae31770d2a314d17bf700433045ca163, entries=200, sequenceid=260, filesize=14.2 K 2023-03-20 10:56:28,571 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:28,571 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:28,572 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,572 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/B/226c26459a444a229fa338b447e9d2b9 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/226c26459a444a229fa338b447e9d2b9 2023-03-20 10:56:28,572 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:28,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:28,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:28,572 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:28,580 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:28,580 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:28,580 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:28,580 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:28,581 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:28,581 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:28,582 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:28,582 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:28,582 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. started... 2023-03-20 10:56:28,582 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. as already flushing 2023-03-20 10:56:28,582 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:28,582 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:28,582 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:28,582 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,582 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:28,582 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:28,584 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,584 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:28,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:28,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:28,585 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:28,585 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:28,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:28,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:28,587 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/226c26459a444a229fa338b447e9d2b9, entries=150, sequenceid=260, filesize=11.9 K 2023-03-20 10:56:28,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:28,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:28,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:28,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3021f37c[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:28,589 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:28,590 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:28,594 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:28,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:28,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:28,594 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:28,594 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,594 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:28,594 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:28,594 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:28,594 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/C/23ba5c831bbc4eddbe9dce7adc592ee2 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/23ba5c831bbc4eddbe9dce7adc592ee2 2023-03-20 10:56:28,597 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,597 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:28,597 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:28,597 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:28,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:28,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:28,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:28,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:28,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:28,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:28,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:28,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:28,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:28,607 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:28,607 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:28,607 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:28,608 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:28,608 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:28,611 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:28,611 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:28,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:28,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:28,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:28,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:28,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:28,611 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:28,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:28,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:28,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:28,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:28,612 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:28,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:28,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:28,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,618 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,618 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/23ba5c831bbc4eddbe9dce7adc592ee2, entries=150, sequenceid=260, filesize=11.9 K 2023-03-20 10:56:28,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:28,622 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for c79c35ba58052cacab6709e8682a60f1 in 1193ms, sequenceid=260, compaction requested=false 2023-03-20 10:56:28,622 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c79c35ba58052cacab6709e8682a60f1: 2023-03-20 10:56:28,623 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,623 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:28,623 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:28,623 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:28,623 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:28,623 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:28,623 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,623 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:28,623 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:28,623 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:28,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:28,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:28,623 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:28,624 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:28,623 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:28,623 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:28,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:28,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:28,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:28,629 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:56:28,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on c79c35ba58052cacab6709e8682a60f1 2023-03-20 10:56:28,639 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c79c35ba58052cacab6709e8682a60f1 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-20 10:56:28,639 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=A 2023-03-20 10:56:28,640 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:28,640 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=B 2023-03-20 10:56:28,640 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:28,640 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=C 2023-03-20 10:56:28,640 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:28,675 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=281 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/A/59c440e99c19402a835f576e27803102 2023-03-20 10:56:28,715 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=281 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/B/3532c85c18de4153ba976b48a0f75655 2023-03-20 10:56:28,719 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:28,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41456 deadline: 1679309848712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,724 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:28,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:28,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:28,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:28,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:28,734 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:28,737 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:28,738 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:28,738 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:28,739 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:28,739 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:28,740 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,740 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:28,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:28,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:28,740 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,740 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:28,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:28,741 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:28,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:28,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:28,741 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:28,742 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:28,742 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:28,742 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:28,742 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:28,743 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:28,744 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:28,744 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:28,744 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:28,745 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. started... 2023-03-20 10:56:28,749 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. as already flushing 2023-03-20 10:56:28,752 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:28,753 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:28,754 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:28,754 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,754 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:28,754 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:28,756 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,756 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:28,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:28,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:28,756 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:28,756 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:28,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:28,757 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:28,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:28,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41436 deadline: 1679309848757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:28,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,758 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=281 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/C/6f92b70c99b2459891ef598cda744076 2023-03-20 10:56:28,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:28,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:28,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35688acb[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:28,758 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:28,758 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:28,759 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:28,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:28,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:28,759 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:28,759 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:28,759 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:28,760 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,760 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:28,762 DEBUG [Thread-145] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4189 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1., hostname=jenkins-hbase20.apache.org,36041,1679309777062, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-20 10:56:28,763 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:28,763 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:28,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:28,763 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:28,769 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:28,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:28,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:28,771 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:28,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:28,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41442 deadline: 1679309848770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,774 DEBUG [Thread-141] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4199 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1., hostname=jenkins-hbase20.apache.org,36041,1679309777062, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-20 10:56:28,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:28,775 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/A/59c440e99c19402a835f576e27803102 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/59c440e99c19402a835f576e27803102 2023-03-20 10:56:28,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:28,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:28,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:28,778 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:28,778 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:28,778 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:28,778 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:28,778 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:28,779 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:28,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:28,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:28,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:28,780 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:28,780 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:28,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:28,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:28,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:28,780 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:28,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:28,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:28,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:28,781 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:28,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:28,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41416 deadline: 1679309848772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:28,782 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:28,793 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/59c440e99c19402a835f576e27803102, entries=200, sequenceid=281, filesize=14.3 K 2023-03-20 10:56:28,785 DEBUG [Thread-143] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4203 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1., hostname=jenkins-hbase20.apache.org,36041,1679309777062, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-20 10:56:28,782 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41388 deadline: 1679309848775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:28,797 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/B/3532c85c18de4153ba976b48a0f75655 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/3532c85c18de4153ba976b48a0f75655 2023-03-20 10:56:28,802 DEBUG [Thread-147] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4226 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1., hostname=jenkins-hbase20.apache.org,36041,1679309777062, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-20 10:56:28,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:28,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,804 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,811 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:28,811 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:28,811 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,811 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:28,811 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:28,811 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:28,805 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:28,813 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:28,813 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:28,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:28,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:28,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:28,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:28,814 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:28,814 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:28,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:28,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:28,816 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:56:28,822 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:28,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41456 deadline: 1679309848822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,844 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/3532c85c18de4153ba976b48a0f75655, entries=150, sequenceid=281, filesize=11.9 K 2023-03-20 10:56:28,847 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/C/6f92b70c99b2459891ef598cda744076 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/6f92b70c99b2459891ef598cda744076 2023-03-20 10:56:28,860 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/6f92b70c99b2459891ef598cda744076, entries=150, sequenceid=281, filesize=11.9 K 2023-03-20 10:56:28,862 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for c79c35ba58052cacab6709e8682a60f1 in 223ms, sequenceid=281, compaction requested=true 2023-03-20 10:56:28,862 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c79c35ba58052cacab6709e8682a60f1: 2023-03-20 10:56:28,862 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:56:28,862 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:56:28,862 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 10:56:28,862 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:56:28,862 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 10:56:28,864 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36564 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 10:56:28,864 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 41394 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 10:56:28,864 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1912): c79c35ba58052cacab6709e8682a60f1/A is initiating minor compaction (all files) 2023-03-20 10:56:28,864 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): c79c35ba58052cacab6709e8682a60f1/B is initiating minor compaction (all files) 2023-03-20 10:56:28,864 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c79c35ba58052cacab6709e8682a60f1/A in TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:28,864 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c79c35ba58052cacab6709e8682a60f1/B in TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:28,864 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/6ee6083833fc437a93190b9610c7ed05, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/ae31770d2a314d17bf700433045ca163, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/59c440e99c19402a835f576e27803102] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp, totalSize=40.4 K 2023-03-20 10:56:28,864 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/b02e7361f3374526b36dcdbcd3ff84d4, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/226c26459a444a229fa338b447e9d2b9, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/3532c85c18de4153ba976b48a0f75655] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp, totalSize=35.7 K 2023-03-20 10:56:28,865 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 6ee6083833fc437a93190b9610c7ed05, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1679309786370 2023-03-20 10:56:28,865 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting b02e7361f3374526b36dcdbcd3ff84d4, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1679309786370 2023-03-20 10:56:28,875 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting ae31770d2a314d17bf700433045ca163, keycount=200, bloomtype=ROW, size=14.2 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1679309787055 2023-03-20 10:56:28,877 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 226c26459a444a229fa338b447e9d2b9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1679309787055 2023-03-20 10:56:28,878 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 59c440e99c19402a835f576e27803102, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=281, earliestPutTs=1679309787499 2023-03-20 10:56:28,879 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 3532c85c18de4153ba976b48a0f75655, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=281, earliestPutTs=1679309787499 2023-03-20 10:56:28,905 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): c79c35ba58052cacab6709e8682a60f1#B#compaction#27 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:56:28,914 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:28,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:28,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:28,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:28,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:28,927 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:28,927 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:28,927 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] throttle.PressureAwareThroughputController(145): c79c35ba58052cacab6709e8682a60f1#A#compaction#28 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:56:28,928 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:28,932 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:28,942 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:28,942 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:28,943 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:28,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:28,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:28,944 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:28,945 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:28,945 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:28,945 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:28,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:28,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:28,945 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:28,950 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:28,950 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:28,951 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:28,951 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:28,970 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:28,978 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:28,978 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:28,981 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:28,982 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. started... 2023-03-20 10:56:28,982 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c79c35ba58052cacab6709e8682a60f1 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-20 10:56:28,982 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=A 2023-03-20 10:56:28,982 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:28,983 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=B 2023-03-20 10:56:28,983 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:28,983 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=C 2023-03-20 10:56:28,983 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:28,998 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/B/f3b4ffa0081646978549f39b6c2429f3 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/f3b4ffa0081646978549f39b6c2429f3 2023-03-20 10:56:28,998 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/A/ec1ba99ed55a4b55befc06b7f94cc590 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/ec1ba99ed55a4b55befc06b7f94cc590 2023-03-20 10:56:29,009 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c79c35ba58052cacab6709e8682a60f1/B of c79c35ba58052cacab6709e8682a60f1 into f3b4ffa0081646978549f39b6c2429f3(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:56:29,011 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c79c35ba58052cacab6709e8682a60f1/A of c79c35ba58052cacab6709e8682a60f1 into ec1ba99ed55a4b55befc06b7f94cc590(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:56:29,011 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c79c35ba58052cacab6709e8682a60f1: 2023-03-20 10:56:29,011 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c79c35ba58052cacab6709e8682a60f1: 2023-03-20 10:56:29,011 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1., storeName=c79c35ba58052cacab6709e8682a60f1/B, priority=13, startTime=1679309788862; duration=0sec 2023-03-20 10:56:29,011 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1., storeName=c79c35ba58052cacab6709e8682a60f1/A, priority=13, startTime=1679309788862; duration=0sec 2023-03-20 10:56:29,012 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:56:29,012 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:56:29,012 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 10:56:29,019 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36564 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 10:56:29,019 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): c79c35ba58052cacab6709e8682a60f1/C is initiating minor compaction (all files) 2023-03-20 10:56:29,019 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c79c35ba58052cacab6709e8682a60f1/C in TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:29,020 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/d9e2ed4b3ac94996b6f925eccc55bcec, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/23ba5c831bbc4eddbe9dce7adc592ee2, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/6f92b70c99b2459891ef598cda744076] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp, totalSize=35.7 K 2023-03-20 10:56:29,021 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting d9e2ed4b3ac94996b6f925eccc55bcec, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1679309786370 2023-03-20 10:56:29,022 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 23ba5c831bbc4eddbe9dce7adc592ee2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1679309787055 2023-03-20 10:56:29,023 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 6f92b70c99b2459891ef598cda744076, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=281, earliestPutTs=1679309787499 2023-03-20 10:56:29,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on c79c35ba58052cacab6709e8682a60f1 2023-03-20 10:56:29,035 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. as already flushing 2023-03-20 10:56:29,052 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): c79c35ba58052cacab6709e8682a60f1#C#compaction#30 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:56:29,075 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/C/4101d396c7f044c7ae6713a99c68c8b4 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/4101d396c7f044c7ae6713a99c68c8b4 2023-03-20 10:56:29,085 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c79c35ba58052cacab6709e8682a60f1/C of c79c35ba58052cacab6709e8682a60f1 into 4101d396c7f044c7ae6713a99c68c8b4(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:56:29,086 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c79c35ba58052cacab6709e8682a60f1: 2023-03-20 10:56:29,086 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1., storeName=c79c35ba58052cacab6709e8682a60f1/C, priority=13, startTime=1679309788862; duration=0sec 2023-03-20 10:56:29,086 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:56:29,129 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:29,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41456 deadline: 1679309849128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:29,232 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:29,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41456 deadline: 1679309849231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:29,434 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:29,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41456 deadline: 1679309849433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:29,454 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=295 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/A/024b7158c3394e8baf3d42e4aee5a35a 2023-03-20 10:56:29,490 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=295 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/B/45246ae816c24193a8df207b20a98452 2023-03-20 10:56:29,528 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=295 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/C/c3125f648e7349dc958d0afb0ee8ff78 2023-03-20 10:56:29,553 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/A/024b7158c3394e8baf3d42e4aee5a35a as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/024b7158c3394e8baf3d42e4aee5a35a 2023-03-20 10:56:29,561 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/024b7158c3394e8baf3d42e4aee5a35a, entries=150, sequenceid=295, filesize=11.9 K 2023-03-20 10:56:29,567 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/B/45246ae816c24193a8df207b20a98452 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/45246ae816c24193a8df207b20a98452 2023-03-20 10:56:29,575 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/45246ae816c24193a8df207b20a98452, entries=150, sequenceid=295, filesize=11.9 K 2023-03-20 10:56:29,580 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/C/c3125f648e7349dc958d0afb0ee8ff78 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/c3125f648e7349dc958d0afb0ee8ff78 2023-03-20 10:56:29,591 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/c3125f648e7349dc958d0afb0ee8ff78, entries=150, sequenceid=295, filesize=11.9 K 2023-03-20 10:56:29,593 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for c79c35ba58052cacab6709e8682a60f1 in 611ms, sequenceid=295, compaction requested=false 2023-03-20 10:56:29,593 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c79c35ba58052cacab6709e8682a60f1: 2023-03-20 10:56:29,593 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:29,593 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:29,593 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:29,593 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:29,593 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:29,593 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:29,595 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:29,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:29,596 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:29,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:29,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:29,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:29,596 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:29,596 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:29,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:29,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:29,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:29,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:29,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:29,602 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61a59ea4[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:29,602 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:29,602 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:29,603 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:29,603 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:29,603 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:29,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:29,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:29,603 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:29,603 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:29,603 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:29,604 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:29,604 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:29,604 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:29,604 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:29,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:29,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:29,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:29,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:29,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:29,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:29,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:29,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:29,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:29,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:29,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:29,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:29,609 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:29,609 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:29,609 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:29,609 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:29,609 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:29,619 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:29,619 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:29,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:29,619 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:29,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:29,619 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:29,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:29,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:29,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:29,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:29,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:29,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:29,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:29,624 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:29,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:29,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:29,627 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:29,627 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:29,627 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:29,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:29,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:29,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:29,627 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:29,628 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:29,628 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:29,628 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:29,628 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:29,627 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:29,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:29,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:29,627 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:29,631 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:56:29,628 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:29,631 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:29,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:29,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:29,731 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:29,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:29,734 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:29,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:29,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:29,742 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:29,742 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:29,742 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:29,742 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:29,742 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:29,743 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:29,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on c79c35ba58052cacab6709e8682a60f1 2023-03-20 10:56:29,743 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c79c35ba58052cacab6709e8682a60f1 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-20 10:56:29,743 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=A 2023-03-20 10:56:29,743 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:29,743 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=B 2023-03-20 10:56:29,743 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:29,744 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=C 2023-03-20 10:56:29,744 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:29,745 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:29,745 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:29,745 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:29,750 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:29,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:29,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:29,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:29,753 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:29,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:29,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:29,754 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:29,755 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:29,755 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:29,755 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:29,755 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:29,769 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:29,769 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:29,769 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:29,769 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. started... 2023-03-20 10:56:29,769 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. as already flushing 2023-03-20 10:56:29,770 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:29,772 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:29,772 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:29,772 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:29,779 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:29,779 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:29,781 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:29,781 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:29,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:29,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:29,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:29,781 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:29,781 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:29,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:29,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:29,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:29,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:29,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:29,784 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:29,784 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55260e1d[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:29,784 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:29,784 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:29,785 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:29,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:29,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:29,786 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:29,786 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:29,786 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:29,786 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:29,786 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:29,787 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:29,787 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:29,787 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:29,787 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:29,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:29,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:29,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:29,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:29,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:29,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:29,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:29,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:29,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:29,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:29,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:29,796 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:29,796 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:29,796 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:29,796 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:29,796 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:29,796 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:29,799 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:29,799 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:29,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:29,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:29,800 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:29,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:29,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:29,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:29,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:29,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:29,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:29,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:29,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:29,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:29,804 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:29,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:29,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:29,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:29,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:29,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:29,807 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:29,812 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:29,812 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:29,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:29,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:29,812 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:29,812 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:29,812 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:29,812 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:29,812 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:29,812 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:29,812 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:29,812 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:29,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:29,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:29,813 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:29,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:29,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:29,813 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:29,816 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:29,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41456 deadline: 1679309849814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:29,817 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=317 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/A/2892c493c58f4e669d30af687e5d4267 2023-03-20 10:56:29,891 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=317 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/B/7476868f0b70473fa287eead7e80a66b 2023-03-20 10:56:29,913 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:29,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:29,921 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:29,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41456 deadline: 1679309849921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:29,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:29,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:29,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:29,927 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:29,927 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:29,927 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:29,927 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:29,928 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:29,928 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:29,929 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:29,929 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:29,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:29,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:29,929 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:29,929 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:29,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:29,930 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:29,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:29,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:29,930 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:29,932 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:29,932 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:29,933 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:29,933 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:29,933 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:29,938 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:29,938 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:29,938 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:29,938 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. started... 2023-03-20 10:56:29,939 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. as already flushing 2023-03-20 10:56:29,939 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:29,939 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:29,939 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:29,939 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:29,966 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:29,966 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:29,970 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:29,970 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:29,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:29,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:29,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:29,971 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:29,971 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:29,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:29,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:29,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:29,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:29,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:29,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:29,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6aa3bdd6[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:29,973 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:29,974 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:29,975 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:29,976 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:29,978 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:29,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:29,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:29,978 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:29,978 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:29,978 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:29,980 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:29,980 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:29,980 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:29,980 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:29,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:29,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:29,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:29,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:29,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:29,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:29,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:29,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:29,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:29,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:29,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:29,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:29,985 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:29,985 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:29,985 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:29,985 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:29,985 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:29,987 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:29,987 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:29,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:29,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:29,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:29,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:29,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:29,990 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:29,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:29,990 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:29,990 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:29,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:29,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:29,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:29,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:29,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:29,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:29,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:29,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:29,995 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:29,995 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:29,995 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:29,995 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:29,995 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:29,996 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:29,996 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:29,996 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:29,996 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:56:29,995 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:29,996 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:29,996 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:29,996 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:29,996 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:29,996 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:29,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:29,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:29,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:29,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:30,096 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:30,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:30,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:30,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:30,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:30,101 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:30,101 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:30,101 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:30,101 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:30,101 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:30,101 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:30,102 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:30,102 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:30,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:30,102 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,102 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:30,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:30,103 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:30,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:30,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:30,103 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:30,103 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:30,103 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:30,103 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:30,103 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:30,103 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:30,106 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:30,106 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:30,106 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:30,106 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. started... 2023-03-20 10:56:30,106 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. as already flushing 2023-03-20 10:56:30,106 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:30,106 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:30,106 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:30,106 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,106 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:30,106 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:30,109 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:30,109 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:30,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:30,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:30,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:30,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:30,112 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:30,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,112 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:30,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:30,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:30,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1bec4c9b[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:30,113 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:30,113 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:30,114 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:30,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:30,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:30,114 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:30,114 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:30,114 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,114 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:30,114 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:30,116 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,116 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:30,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:30,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:30,116 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:30,116 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:30,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:30,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:30,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:30,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:30,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:30,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:30,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:30,118 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:30,118 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:30,118 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:30,118 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:30,118 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:30,119 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:30,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:30,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:30,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:30,120 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:30,120 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:30,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:30,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:30,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:30,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:30,120 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:30,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:30,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:30,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:30,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:30,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,121 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:30,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:30,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,126 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:30,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41456 deadline: 1679309850125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,128 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,128 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:30,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:30,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:30,129 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:56:30,128 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:30,129 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:30,129 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,129 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:30,129 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:30,129 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:30,129 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:30,129 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:30,129 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:30,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:30,129 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:30,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:30,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:30,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:30,229 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:30,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:30,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:30,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:30,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:30,246 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:30,246 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:30,246 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:30,246 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:30,247 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:30,247 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:30,248 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,250 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,250 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:30,251 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:30,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:30,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:30,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:30,252 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:30,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:30,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:30,253 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:30,253 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:30,253 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:30,253 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:30,253 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:30,253 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:30,253 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:30,253 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:30,253 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:30,253 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. started... 2023-03-20 10:56:30,253 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. as already flushing 2023-03-20 10:56:30,253 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:30,254 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:30,254 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:30,254 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,254 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:30,254 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:30,260 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,260 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:30,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:30,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:30,261 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:30,261 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:30,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:30,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:30,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:30,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:30,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:30,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5671b1fc[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:30,262 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:30,262 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:30,263 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:30,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:30,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:30,263 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:30,263 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:30,263 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:30,264 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,264 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:30,264 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:30,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:30,265 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:30,265 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:30,265 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:30,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:30,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:30,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:30,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:30,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:30,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:30,269 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:30,269 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:30,270 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:30,270 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:30,270 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:30,270 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:30,279 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:30,279 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:30,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:30,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:30,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:30,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:30,279 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:30,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:30,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:30,280 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:30,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:30,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:30,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:30,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:30,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:30,282 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:30,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:30,285 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:30,285 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:30,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:30,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:30,286 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,286 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:30,286 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:30,286 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,286 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:30,286 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:30,286 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:30,286 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:30,286 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:30,286 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:30,286 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:30,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:30,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:30,286 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:30,287 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:56:30,375 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=317 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/C/b6490a7fd52c4005b6c607d9a300f7d5 2023-03-20 10:56:30,383 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/A/2892c493c58f4e669d30af687e5d4267 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/2892c493c58f4e669d30af687e5d4267 2023-03-20 10:56:30,386 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:30,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:30,390 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/2892c493c58f4e669d30af687e5d4267, entries=200, sequenceid=317, filesize=14.3 K 2023-03-20 10:56:30,393 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/B/7476868f0b70473fa287eead7e80a66b as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/7476868f0b70473fa287eead7e80a66b 2023-03-20 10:56:30,395 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:30,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:30,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:30,397 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:30,397 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:30,397 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:30,397 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:30,397 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:30,397 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:30,398 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:30,398 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:30,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:30,398 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,398 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:30,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:30,399 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:30,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:30,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:30,400 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:30,401 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/7476868f0b70473fa287eead7e80a66b, entries=150, sequenceid=317, filesize=11.9 K 2023-03-20 10:56:30,408 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:30,408 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:30,408 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:30,408 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:30,408 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/C/b6490a7fd52c4005b6c607d9a300f7d5 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/b6490a7fd52c4005b6c607d9a300f7d5 2023-03-20 10:56:30,408 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:30,409 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:30,409 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:30,409 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:30,409 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. started... 2023-03-20 10:56:30,409 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. as already flushing 2023-03-20 10:56:30,409 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:30,409 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:30,409 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:30,409 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,409 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:30,409 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:30,411 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:30,411 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:30,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:30,414 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:30,414 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:30,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:30,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:30,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:30,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:30,416 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:30,416 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33c20bc9[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:30,416 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:30,416 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:30,418 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/b6490a7fd52c4005b6c607d9a300f7d5, entries=150, sequenceid=317, filesize=11.9 K 2023-03-20 10:56:30,419 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:30,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:30,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:30,419 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:30,419 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:30,419 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:30,419 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,420 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:30,420 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for c79c35ba58052cacab6709e8682a60f1 in 676ms, sequenceid=317, compaction requested=true 2023-03-20 10:56:30,420 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c79c35ba58052cacab6709e8682a60f1: 2023-03-20 10:56:30,420 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:56:30,420 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 10:56:30,420 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:56:30,420 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-20 10:56:30,420 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:30,420 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:30,420 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:30,421 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39306 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 10:56:30,421 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1912): c79c35ba58052cacab6709e8682a60f1/A is initiating minor compaction (all files) 2023-03-20 10:56:30,422 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c79c35ba58052cacab6709e8682a60f1/A in TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:30,422 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/ec1ba99ed55a4b55befc06b7f94cc590, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/024b7158c3394e8baf3d42e4aee5a35a, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/2892c493c58f4e669d30af687e5d4267] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp, totalSize=38.4 K 2023-03-20 10:56:30,422 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 10:56:30,422 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:30,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:30,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:30,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:30,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:30,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,424 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting ec1ba99ed55a4b55befc06b7f94cc590, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=281, earliestPutTs=1679309787499 2023-03-20 10:56:30,425 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 024b7158c3394e8baf3d42e4aee5a35a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=295, earliestPutTs=1679309788675 2023-03-20 10:56:30,425 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36866 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 10:56:30,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:30,425 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): c79c35ba58052cacab6709e8682a60f1/B is initiating minor compaction (all files) 2023-03-20 10:56:30,426 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c79c35ba58052cacab6709e8682a60f1/B in TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:30,426 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/f3b4ffa0081646978549f39b6c2429f3, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/45246ae816c24193a8df207b20a98452, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/7476868f0b70473fa287eead7e80a66b] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp, totalSize=36.0 K 2023-03-20 10:56:30,426 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 2892c493c58f4e669d30af687e5d4267, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1679309789085 2023-03-20 10:56:30,427 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting f3b4ffa0081646978549f39b6c2429f3, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=281, earliestPutTs=1679309787499 2023-03-20 10:56:30,428 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 45246ae816c24193a8df207b20a98452, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=295, earliestPutTs=1679309788675 2023-03-20 10:56:30,428 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 7476868f0b70473fa287eead7e80a66b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1679309789085 2023-03-20 10:56:30,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:30,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:30,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:30,433 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:30,433 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:30,433 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:30,433 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:30,433 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:30,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on c79c35ba58052cacab6709e8682a60f1 2023-03-20 10:56:30,435 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c79c35ba58052cacab6709e8682a60f1 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-20 10:56:30,435 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=A 2023-03-20 10:56:30,435 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:30,435 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=B 2023-03-20 10:56:30,435 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:30,435 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=C 2023-03-20 10:56:30,435 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:30,442 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] throttle.PressureAwareThroughputController(145): c79c35ba58052cacab6709e8682a60f1#A#compaction#36 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:56:30,443 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:30,443 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:30,443 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:30,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:30,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:30,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:30,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:30,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:30,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:30,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:30,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:30,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:30,446 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:30,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:30,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:30,449 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:30,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:30,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,452 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:30,452 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:30,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:30,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:30,452 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,452 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:30,452 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:30,453 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:30,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:30,453 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:30,453 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:30,453 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:30,453 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:30,453 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:56:30,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:30,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:30,453 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:30,453 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:30,467 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): c79c35ba58052cacab6709e8682a60f1#B#compaction#38 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:56:30,488 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=331 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/A/a7b96586d19f411291abab7e85b7e51c 2023-03-20 10:56:30,533 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/A/21f87be3656a4eab8d7d3ee98676a8c6 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/21f87be3656a4eab8d7d3ee98676a8c6 2023-03-20 10:56:30,553 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:30,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:30,555 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:30,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41456 deadline: 1679309850553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:30,566 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c79c35ba58052cacab6709e8682a60f1/A of c79c35ba58052cacab6709e8682a60f1 into 21f87be3656a4eab8d7d3ee98676a8c6(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:56:30,566 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c79c35ba58052cacab6709e8682a60f1: 2023-03-20 10:56:30,567 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1., storeName=c79c35ba58052cacab6709e8682a60f1/A, priority=13, startTime=1679309790420; duration=0sec 2023-03-20 10:56:30,567 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:56:30,567 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 10:56:30,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:30,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:30,567 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:30,567 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:30,567 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:30,567 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:30,568 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:30,568 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:30,569 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:30,569 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:30,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:30,569 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=331 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/B/04a0d2f7ceaf48ddac851170d168e83c 2023-03-20 10:56:30,569 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,569 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:30,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:30,570 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36866 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 10:56:30,570 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1912): c79c35ba58052cacab6709e8682a60f1/C is initiating minor compaction (all files) 2023-03-20 10:56:30,570 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c79c35ba58052cacab6709e8682a60f1/C in TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:30,570 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:30,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:30,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:30,570 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:30,571 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/4101d396c7f044c7ae6713a99c68c8b4, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/c3125f648e7349dc958d0afb0ee8ff78, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/b6490a7fd52c4005b6c607d9a300f7d5] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp, totalSize=36.0 K 2023-03-20 10:56:30,571 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:30,571 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:30,571 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:30,571 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:30,571 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:30,572 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:30,572 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:30,572 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:30,572 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 4101d396c7f044c7ae6713a99c68c8b4, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=281, earliestPutTs=1679309787499 2023-03-20 10:56:30,572 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. started... 2023-03-20 10:56:30,572 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. as already flushing 2023-03-20 10:56:30,572 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:30,572 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:30,572 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:30,572 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,572 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:30,572 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:30,576 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting c3125f648e7349dc958d0afb0ee8ff78, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=295, earliestPutTs=1679309788675 2023-03-20 10:56:30,577 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting b6490a7fd52c4005b6c607d9a300f7d5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1679309789085 2023-03-20 10:56:30,580 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,580 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:30,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:30,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:30,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:30,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:30,581 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:30,581 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:30,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:30,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:30,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:30,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44785d6a[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:30,582 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:30,582 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:30,584 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:30,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:30,584 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,584 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:30,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:30,585 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:30,585 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:30,585 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:30,586 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:30,586 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:30,586 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:30,586 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:30,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:30,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:30,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:30,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:30,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:30,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:30,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:30,593 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:30,593 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:30,593 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:30,593 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:30,593 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:30,593 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:30,595 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:30,595 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:30,595 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:30,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:30,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:30,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:30,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:30,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:30,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:30,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:30,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:30,596 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:30,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:30,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:30,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:30,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,598 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:30,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:30,599 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] throttle.PressureAwareThroughputController(145): c79c35ba58052cacab6709e8682a60f1#C#compaction#41 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:56:30,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,610 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:30,610 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:30,610 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:56:30,610 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:30,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:30,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:30,610 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:30,610 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:30,610 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,610 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:30,610 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:30,610 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:30,610 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:30,611 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:30,611 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:30,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:30,611 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=331 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/C/1e0b26e416cb48deb3f9c94376e47a8c 2023-03-20 10:56:30,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:30,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:30,620 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/A/a7b96586d19f411291abab7e85b7e51c as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/a7b96586d19f411291abab7e85b7e51c 2023-03-20 10:56:30,624 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/C/07435a7fb05a4f9ca5f789352a4695ae as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/07435a7fb05a4f9ca5f789352a4695ae 2023-03-20 10:56:30,631 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/a7b96586d19f411291abab7e85b7e51c, entries=200, sequenceid=331, filesize=14.3 K 2023-03-20 10:56:30,634 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/B/04a0d2f7ceaf48ddac851170d168e83c as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/04a0d2f7ceaf48ddac851170d168e83c 2023-03-20 10:56:30,637 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c79c35ba58052cacab6709e8682a60f1/C of c79c35ba58052cacab6709e8682a60f1 into 07435a7fb05a4f9ca5f789352a4695ae(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:56:30,637 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c79c35ba58052cacab6709e8682a60f1: 2023-03-20 10:56:30,637 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1., storeName=c79c35ba58052cacab6709e8682a60f1/C, priority=13, startTime=1679309790420; duration=0sec 2023-03-20 10:56:30,637 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:56:30,646 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/04a0d2f7ceaf48ddac851170d168e83c, entries=150, sequenceid=331, filesize=11.9 K 2023-03-20 10:56:30,647 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/C/1e0b26e416cb48deb3f9c94376e47a8c as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/1e0b26e416cb48deb3f9c94376e47a8c 2023-03-20 10:56:30,657 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/1e0b26e416cb48deb3f9c94376e47a8c, entries=150, sequenceid=331, filesize=11.9 K 2023-03-20 10:56:30,658 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for c79c35ba58052cacab6709e8682a60f1 in 223ms, sequenceid=331, compaction requested=false 2023-03-20 10:56:30,658 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c79c35ba58052cacab6709e8682a60f1: 2023-03-20 10:56:30,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on c79c35ba58052cacab6709e8682a60f1 2023-03-20 10:56:30,674 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c79c35ba58052cacab6709e8682a60f1 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-20 10:56:30,674 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=A 2023-03-20 10:56:30,675 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:30,675 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=B 2023-03-20 10:56:30,675 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:30,675 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=C 2023-03-20 10:56:30,675 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:30,711 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:30,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:30,714 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=353 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/A/edbfd599255848df8334028b263c46a1 2023-03-20 10:56:30,715 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:30,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41456 deadline: 1679309850713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:30,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:30,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:30,722 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:30,722 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:30,722 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:30,722 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:30,723 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:30,723 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:30,724 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,724 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:30,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:30,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:30,725 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,725 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:30,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:30,726 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:30,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:30,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:30,727 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:30,727 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:30,727 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:30,728 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:30,728 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:30,728 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:30,728 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:30,729 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:30,729 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:30,729 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. started... 2023-03-20 10:56:30,729 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. as already flushing 2023-03-20 10:56:30,729 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:30,729 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:30,729 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:30,729 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,729 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:30,729 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:30,736 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:30,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:30,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:30,737 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:30,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:30,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:30,738 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:30,738 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:30,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:30,740 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:30,740 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d0949ea[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:30,740 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:30,741 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:30,743 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:30,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:30,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:30,744 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:30,744 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:30,744 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:30,744 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,744 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:30,746 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:30,746 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,746 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:30,746 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:30,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:30,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:30,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:30,747 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=353 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/B/72c76ad14a7140269a2173d3f6ad3180 2023-03-20 10:56:30,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:30,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:30,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:30,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:30,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:30,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:30,751 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:30,751 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:30,751 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:30,751 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:30,751 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:30,753 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:30,753 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:30,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:30,753 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:30,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:30,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:30,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:30,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:30,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:30,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:30,753 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:30,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:30,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:30,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:30,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:30,755 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:30,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:30,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,760 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,761 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:30,761 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:30,761 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:30,761 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,761 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:30,761 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:30,761 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:30,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:30,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:30,761 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:56:30,761 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:30,761 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:30,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:30,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:30,761 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:30,762 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:30,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:30,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:30,825 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:30,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41456 deadline: 1679309850823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,862 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:30,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:30,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:30,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:30,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:30,876 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:30,877 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:30,877 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:30,877 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:30,880 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:30,880 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:30,884 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:30,884 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:30,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:30,890 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,890 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:30,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:30,891 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:30,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:30,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:30,894 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:30,894 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:30,894 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:30,897 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:30,897 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:30,897 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:30,898 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:30,898 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:30,898 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:30,898 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. started... 2023-03-20 10:56:30,898 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. as already flushing 2023-03-20 10:56:30,898 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:30,898 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:30,898 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:30,898 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,898 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:30,899 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:30,902 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,902 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:30,902 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:30,902 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:30,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:30,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:30,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:30,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:30,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:30,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:30,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:30,934 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:30,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@662cf1e6[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:30,934 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:30,937 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:30,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:30,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:30,937 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,937 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:30,937 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:30,937 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:30,937 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:30,939 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:30,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:30,939 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:30,939 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:30,939 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:30,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:30,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:30,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:30,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:30,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:30,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:30,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:30,943 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:30,944 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:30,944 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:30,944 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:30,944 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:30,945 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:30,945 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:30,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:30,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:30,945 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:30,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:30,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:30,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:30,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:30,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:30,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:30,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:30,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:30,950 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:30,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:30,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,954 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/B/4f521188449745e0b8fbbf0279406775 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/4f521188449745e0b8fbbf0279406775 2023-03-20 10:56:30,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:30,955 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:30,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,960 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:30,961 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:30,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:30,961 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:30,961 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:30,961 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:30,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:30,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:30,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:30,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:30,961 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:30,961 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:30,961 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:30,961 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:30,962 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:30,962 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:30,962 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:30,966 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c79c35ba58052cacab6709e8682a60f1/B of c79c35ba58052cacab6709e8682a60f1 into 4f521188449745e0b8fbbf0279406775(size=12.3 K), total size for store is 24.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:56:30,966 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c79c35ba58052cacab6709e8682a60f1: 2023-03-20 10:56:30,966 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1., storeName=c79c35ba58052cacab6709e8682a60f1/B, priority=13, startTime=1679309790420; duration=0sec 2023-03-20 10:56:30,966 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:56:31,031 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:31,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41456 deadline: 1679309851028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,061 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:31,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:31,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:31,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:31,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:31,065 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:31,065 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:31,065 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:31,065 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:31,065 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:31,065 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:31,066 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:31,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:31,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:31,066 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:31,066 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,066 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:31,067 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:31,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:31,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:31,067 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:31,067 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:31,067 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:31,067 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:31,067 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:31,067 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:31,068 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:31,068 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:31,068 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:31,068 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. started... 2023-03-20 10:56:31,068 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. as already flushing 2023-03-20 10:56:31,068 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:31,068 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:31,068 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:31,068 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,068 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:31,068 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:31,069 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,069 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:31,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:31,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:31,069 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:31,069 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:31,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:31,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:31,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:31,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:31,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:31,071 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:31,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@451322a9[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:31,071 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:31,071 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:31,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:31,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:31,071 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:31,072 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:31,072 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:31,072 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,072 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:31,072 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,072 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:31,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:31,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:31,073 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:31,073 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:31,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:31,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:31,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:31,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:31,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:31,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:31,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:31,074 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:31,074 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:31,074 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:31,074 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:31,075 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:31,076 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:31,076 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:31,076 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:31,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:31,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:31,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:31,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:31,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:31,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:31,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:31,076 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:31,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:31,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:31,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:31,078 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:31,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:31,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:31,080 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:31,080 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:31,080 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:31,080 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:31,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:31,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:31,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:31,080 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:31,080 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:31,080 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:31,080 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:31,080 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:31,080 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:31,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:31,081 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:31,081 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:31,081 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:56:31,169 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=353 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/C/75e6c15dee4c40748e632457edd1e209 2023-03-20 10:56:31,176 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/A/edbfd599255848df8334028b263c46a1 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/edbfd599255848df8334028b263c46a1 2023-03-20 10:56:31,180 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:31,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:31,183 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/edbfd599255848df8334028b263c46a1, entries=200, sequenceid=353, filesize=14.3 K 2023-03-20 10:56:31,184 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:31,184 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/B/72c76ad14a7140269a2173d3f6ad3180 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/72c76ad14a7140269a2173d3f6ad3180 2023-03-20 10:56:31,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:31,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:31,187 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:31,187 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:31,187 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:31,187 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:31,188 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:31,188 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:31,189 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:31,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:31,189 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:31,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:31,190 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,190 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:31,190 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:31,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:31,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:31,190 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:31,191 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:31,191 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:31,191 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:31,191 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:31,191 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:31,192 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:31,192 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:31,192 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:31,192 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. started... 2023-03-20 10:56:31,192 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. as already flushing 2023-03-20 10:56:31,192 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:31,192 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:31,192 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:31,192 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,192 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:31,192 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:31,202 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,202 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:31,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:31,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:31,203 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:31,203 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:31,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:31,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:31,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:31,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:31,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:31,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4306a46c[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:31,205 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:31,205 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:31,206 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:31,206 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,206 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:31,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:31,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:31,206 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:31,206 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:31,206 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:31,207 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/72c76ad14a7140269a2173d3f6ad3180, entries=150, sequenceid=353, filesize=11.9 K 2023-03-20 10:56:31,209 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/C/75e6c15dee4c40748e632457edd1e209 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/75e6c15dee4c40748e632457edd1e209 2023-03-20 10:56:31,219 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,219 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:31,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:31,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:31,219 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:31,219 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:31,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:31,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:31,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:31,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:31,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:31,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,223 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/75e6c15dee4c40748e632457edd1e209, entries=150, sequenceid=353, filesize=11.9 K 2023-03-20 10:56:31,224 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for c79c35ba58052cacab6709e8682a60f1 in 550ms, sequenceid=353, compaction requested=true 2023-03-20 10:56:31,224 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c79c35ba58052cacab6709e8682a60f1: 2023-03-20 10:56:31,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:31,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:31,224 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:31,224 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:31,225 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:31,225 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:56:31,224 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 10:56:31,225 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:56:31,225 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 10:56:31,225 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:31,225 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:56:31,225 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:31,226 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 41848 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 10:56:31,226 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36968 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 10:56:31,226 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1912): c79c35ba58052cacab6709e8682a60f1/A is initiating minor compaction (all files) 2023-03-20 10:56:31,227 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): c79c35ba58052cacab6709e8682a60f1/B is initiating minor compaction (all files) 2023-03-20 10:56:31,227 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c79c35ba58052cacab6709e8682a60f1/A in TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:31,227 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c79c35ba58052cacab6709e8682a60f1/B in TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:31,227 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/21f87be3656a4eab8d7d3ee98676a8c6, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/a7b96586d19f411291abab7e85b7e51c, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/edbfd599255848df8334028b263c46a1] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp, totalSize=40.9 K 2023-03-20 10:56:31,227 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/4f521188449745e0b8fbbf0279406775, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/04a0d2f7ceaf48ddac851170d168e83c, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/72c76ad14a7140269a2173d3f6ad3180] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp, totalSize=36.1 K 2023-03-20 10:56:31,227 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:31,227 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:31,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:31,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:31,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:31,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:31,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:31,227 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:31,228 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 21f87be3656a4eab8d7d3ee98676a8c6, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1679309789085 2023-03-20 10:56:31,228 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 4f521188449745e0b8fbbf0279406775, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1679309789085 2023-03-20 10:56:31,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:31,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:31,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:31,228 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,228 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting a7b96586d19f411291abab7e85b7e51c, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=331, earliestPutTs=1679309789796 2023-03-20 10:56:31,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:31,228 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 04a0d2f7ceaf48ddac851170d168e83c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=331, earliestPutTs=1679309789796 2023-03-20 10:56:31,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:31,229 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting edbfd599255848df8334028b263c46a1, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=353, earliestPutTs=1679309790544 2023-03-20 10:56:31,229 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 72c76ad14a7140269a2173d3f6ad3180, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=353, earliestPutTs=1679309790544 2023-03-20 10:56:31,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:31,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:31,230 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:31,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:31,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,236 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:31,236 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:31,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:31,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:31,237 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,237 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:31,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:31,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:31,237 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:31,237 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:31,237 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:31,237 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,237 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:31,237 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:31,238 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:31,238 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:31,242 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:56:31,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:31,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:31,250 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] throttle.PressureAwareThroughputController(145): c79c35ba58052cacab6709e8682a60f1#A#compaction#45 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:56:31,253 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): c79c35ba58052cacab6709e8682a60f1#B#compaction#46 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:56:31,306 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/B/cdeedb8f493848b0a6d4169dfda1f961 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/cdeedb8f493848b0a6d4169dfda1f961 2023-03-20 10:56:31,317 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c79c35ba58052cacab6709e8682a60f1/B of c79c35ba58052cacab6709e8682a60f1 into cdeedb8f493848b0a6d4169dfda1f961(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:56:31,318 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c79c35ba58052cacab6709e8682a60f1: 2023-03-20 10:56:31,318 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1., storeName=c79c35ba58052cacab6709e8682a60f1/B, priority=13, startTime=1679309791225; duration=0sec 2023-03-20 10:56:31,318 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:56:31,318 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 10:56:31,334 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36968 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 10:56:31,334 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): c79c35ba58052cacab6709e8682a60f1/C is initiating minor compaction (all files) 2023-03-20 10:56:31,334 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c79c35ba58052cacab6709e8682a60f1/C in TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:31,334 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/07435a7fb05a4f9ca5f789352a4695ae, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/1e0b26e416cb48deb3f9c94376e47a8c, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/75e6c15dee4c40748e632457edd1e209] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp, totalSize=36.1 K 2023-03-20 10:56:31,336 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 07435a7fb05a4f9ca5f789352a4695ae, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1679309789085 2023-03-20 10:56:31,337 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 1e0b26e416cb48deb3f9c94376e47a8c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=331, earliestPutTs=1679309789796 2023-03-20 10:56:31,337 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 75e6c15dee4c40748e632457edd1e209, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=353, earliestPutTs=1679309790544 2023-03-20 10:56:31,338 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:31,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:31,341 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c79c35ba58052cacab6709e8682a60f1 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-20 10:56:31,341 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=A 2023-03-20 10:56:31,341 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:31,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on c79c35ba58052cacab6709e8682a60f1 2023-03-20 10:56:31,341 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=B 2023-03-20 10:56:31,341 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:31,341 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=C 2023-03-20 10:56:31,342 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:31,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:31,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:31,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:31,359 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:31,359 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:31,359 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:31,359 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:31,359 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:31,359 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:31,363 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:31,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:31,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:31,364 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:31,364 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,364 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:31,365 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:31,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:31,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:31,365 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:31,372 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:31,372 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:31,372 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:31,372 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:31,372 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:31,374 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:31,374 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:31,376 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:31,376 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): c79c35ba58052cacab6709e8682a60f1#C#compaction#48 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:56:31,376 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. started... 2023-03-20 10:56:31,382 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. as already flushing 2023-03-20 10:56:31,382 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:31,382 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:31,382 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:31,382 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,382 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:31,382 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:31,386 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:31,386 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:31,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:31,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:31,387 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:31,388 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:31,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:31,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:31,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:31,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:31,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34b7f3ff[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:31,391 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:31,391 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:31,397 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/C/12919e9f7e1e4b91904908a980b56be4 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/12919e9f7e1e4b91904908a980b56be4 2023-03-20 10:56:31,400 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:31,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:31,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:31,400 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:31,400 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:31,400 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:31,401 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,401 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:31,410 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:31,411 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:31,411 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:31,411 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:31,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:31,412 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=368 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/A/c709adc09d36496e941d6f522cc07b84 2023-03-20 10:56:31,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:31,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:31,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:31,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:31,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:31,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:31,416 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:31,416 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:31,416 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:31,416 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:31,416 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:31,416 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:31,420 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:31,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:31,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:31,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:31,420 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:31,420 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:31,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:31,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:31,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:31,421 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:31,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:31,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:31,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:31,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:31,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:31,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,423 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c79c35ba58052cacab6709e8682a60f1/C of c79c35ba58052cacab6709e8682a60f1 into 12919e9f7e1e4b91904908a980b56be4(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:56:31,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:31,423 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c79c35ba58052cacab6709e8682a60f1: 2023-03-20 10:56:31,423 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1., storeName=c79c35ba58052cacab6709e8682a60f1/C, priority=13, startTime=1679309791225; duration=0sec 2023-03-20 10:56:31,423 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,423 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:56:31,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:31,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,427 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,427 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:31,427 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:31,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:31,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:31,427 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:31,428 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:31,428 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,428 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:31,428 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:31,428 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:31,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:31,428 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:31,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:31,428 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:31,428 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:31,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:31,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:31,489 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:31,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41456 deadline: 1679309851484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,528 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:31,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:31,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:31,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:31,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:31,537 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:31,538 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:31,538 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:31,538 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:31,539 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:31,539 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:31,541 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,541 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:31,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:31,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:31,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:31,542 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,542 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:31,542 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:31,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:31,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:31,543 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:31,546 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:31,546 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:31,546 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:31,546 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:31,546 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:31,546 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:31,546 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:31,546 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:31,547 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. started... 2023-03-20 10:56:31,547 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. as already flushing 2023-03-20 10:56:31,547 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:31,547 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:31,547 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:31,547 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,547 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:31,547 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:31,552 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,552 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:31,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:31,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:31,552 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:31,552 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:31,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:31,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:31,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:31,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:31,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:31,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3515874e[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:31,554 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:31,554 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:31,560 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:31,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:31,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:31,561 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:31,561 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,561 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:31,561 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:31,561 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:31,568 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:31,568 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,568 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:31,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,569 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:31,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:31,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:31,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:31,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:31,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:31,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:31,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:31,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:31,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:31,575 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:31,575 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:31,575 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:31,575 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:31,575 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:31,577 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:31,577 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:31,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:31,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:31,578 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:31,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:31,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:31,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:31,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:31,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:31,580 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:31,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:31,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:31,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:31,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:31,581 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:31,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:31,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,586 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,586 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:31,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:31,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:31,586 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:31,586 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:31,586 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,586 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:31,586 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:31,586 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:31,587 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:31,587 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:31,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:31,586 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:31,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:31,587 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:31,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:31,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:31,592 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:31,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41456 deadline: 1679309851590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,687 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:31,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:31,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:31,692 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/A/19e5b29188d243bd9aec3d472c4d1131 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/19e5b29188d243bd9aec3d472c4d1131 2023-03-20 10:56:31,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:31,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:31,695 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:31,695 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:31,695 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:31,695 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:31,698 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:31,699 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:31,699 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,699 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:31,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:31,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:31,700 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,700 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:31,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:31,700 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:31,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:31,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:31,701 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:31,704 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c79c35ba58052cacab6709e8682a60f1/A of c79c35ba58052cacab6709e8682a60f1 into 19e5b29188d243bd9aec3d472c4d1131(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:56:31,704 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c79c35ba58052cacab6709e8682a60f1: 2023-03-20 10:56:31,704 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1., storeName=c79c35ba58052cacab6709e8682a60f1/A, priority=13, startTime=1679309791224; duration=0sec 2023-03-20 10:56:31,704 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:56:31,706 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:31,706 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:31,706 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:31,706 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:31,706 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:31,706 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:31,706 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:31,706 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:31,706 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. started... 2023-03-20 10:56:31,706 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. as already flushing 2023-03-20 10:56:31,707 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:31,707 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:31,707 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:31,707 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,707 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:31,707 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:31,710 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:31,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:31,710 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:31,711 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:31,711 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:31,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:31,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:31,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:31,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:31,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:31,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@694ffd2f[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:31,713 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:31,713 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:31,714 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:31,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:31,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:31,714 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:31,714 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:31,714 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:31,714 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,714 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:31,715 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,715 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:31,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:31,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:31,715 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:31,715 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:31,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:31,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:31,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:31,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:31,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:31,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:31,717 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:31,717 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:31,718 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:31,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:31,718 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:31,718 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:31,719 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:31,719 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:31,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:31,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:31,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:31,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:31,719 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:31,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:31,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:31,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:31,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:31,720 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:31,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:31,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:31,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:31,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,721 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:31,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:31,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,723 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:31,723 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,723 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:31,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:31,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:31,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:31,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:31,723 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:31,723 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:31,724 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:31,724 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,724 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:31,724 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:31,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:31,724 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:31,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:31,724 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:31,724 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:31,796 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:31,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41456 deadline: 1679309851793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,824 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:31,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:31,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:31,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:31,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:31,827 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:31,827 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:31,828 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:31,828 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:31,828 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:31,828 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:31,829 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:31,829 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:31,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:31,829 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,829 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:31,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:31,830 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:31,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:31,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:31,830 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:31,831 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:31,831 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:31,831 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:31,831 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:31,831 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:31,831 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:31,831 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:31,831 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:31,831 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. started... 2023-03-20 10:56:31,831 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. as already flushing 2023-03-20 10:56:31,831 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:31,831 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:31,831 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:31,831 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,831 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:31,831 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:31,832 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:31,832 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:31,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:31,833 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:31,833 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:31,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:31,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:31,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:31,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:31,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:31,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f1c01ae[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:31,834 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:31,834 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:31,835 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:31,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:31,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:31,835 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:31,835 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:31,835 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:31,836 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,836 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:31,838 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,838 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:31,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:31,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:31,838 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:31,838 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:31,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:31,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:31,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:31,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:31,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:31,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:31,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:31,841 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:31,841 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:31,841 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:31,841 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:31,841 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:31,844 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:31,844 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:31,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:31,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:31,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:31,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:31,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:31,844 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:31,845 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:31,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:31,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:31,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:31,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:31,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:31,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:31,848 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:31,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:31,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,861 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:31,861 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:31,861 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:31,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:31,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:31,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:31,861 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:31,861 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:31,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:31,861 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,862 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:56:31,862 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:31,862 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:31,862 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,862 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:31,862 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:31,862 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:31,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:31,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:31,869 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=368 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/B/ff7e43113ce64683a932a0811942de17 2023-03-20 10:56:31,923 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=368 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/C/977f0a046b504392896f1de49b3d7d6f 2023-03-20 10:56:31,936 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/A/c709adc09d36496e941d6f522cc07b84 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/c709adc09d36496e941d6f522cc07b84 2023-03-20 10:56:31,949 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/c709adc09d36496e941d6f522cc07b84, entries=200, sequenceid=368, filesize=14.3 K 2023-03-20 10:56:31,950 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/B/ff7e43113ce64683a932a0811942de17 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/ff7e43113ce64683a932a0811942de17 2023-03-20 10:56:31,958 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/ff7e43113ce64683a932a0811942de17, entries=150, sequenceid=368, filesize=11.9 K 2023-03-20 10:56:31,961 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:31,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:31,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:31,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:31,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:31,968 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:31,968 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:31,968 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:31,968 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:31,969 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/C/977f0a046b504392896f1de49b3d7d6f as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/977f0a046b504392896f1de49b3d7d6f 2023-03-20 10:56:31,970 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:31,970 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:31,971 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,971 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:31,972 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:31,973 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:31,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:31,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:31,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:31,976 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/977f0a046b504392896f1de49b3d7d6f, entries=150, sequenceid=368, filesize=11.9 K 2023-03-20 10:56:31,976 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:31,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:31,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:31,976 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:31,977 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for c79c35ba58052cacab6709e8682a60f1 in 636ms, sequenceid=368, compaction requested=false 2023-03-20 10:56:31,977 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:31,977 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c79c35ba58052cacab6709e8682a60f1: 2023-03-20 10:56:31,977 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:31,977 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:31,977 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:31,977 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:31,977 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:31,977 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:31,977 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. started... 2023-03-20 10:56:31,977 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:31,978 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c79c35ba58052cacab6709e8682a60f1 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-20 10:56:31,978 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=A 2023-03-20 10:56:31,978 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:31,978 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=B 2023-03-20 10:56:31,978 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:31,978 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=C 2023-03-20 10:56:31,978 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:31,992 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=389 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/A/76794a01628e4aa5b96ad3cc301cbe32 2023-03-20 10:56:32,012 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=389 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/B/682afc884e684f7c8794383ebb8cdbb1 2023-03-20 10:56:32,051 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=389 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/C/e9b8cc68bbc1490bb6f47317a6e320a3 2023-03-20 10:56:32,064 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/A/76794a01628e4aa5b96ad3cc301cbe32 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/76794a01628e4aa5b96ad3cc301cbe32 2023-03-20 10:56:32,083 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/76794a01628e4aa5b96ad3cc301cbe32, entries=150, sequenceid=389, filesize=11.9 K 2023-03-20 10:56:32,085 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/B/682afc884e684f7c8794383ebb8cdbb1 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/682afc884e684f7c8794383ebb8cdbb1 2023-03-20 10:56:32,094 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/682afc884e684f7c8794383ebb8cdbb1, entries=150, sequenceid=389, filesize=11.9 K 2023-03-20 10:56:32,095 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/C/e9b8cc68bbc1490bb6f47317a6e320a3 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/e9b8cc68bbc1490bb6f47317a6e320a3 2023-03-20 10:56:32,122 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/e9b8cc68bbc1490bb6f47317a6e320a3, entries=150, sequenceid=389, filesize=11.9 K 2023-03-20 10:56:32,124 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=6.71 KB/6870 for c79c35ba58052cacab6709e8682a60f1 in 146ms, sequenceid=389, compaction requested=true 2023-03-20 10:56:32,124 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c79c35ba58052cacab6709e8682a60f1: 2023-03-20 10:56:32,124 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:32,124 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:32,124 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:32,124 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,124 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:32,124 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:32,125 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:32,130 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:32,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:32,130 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:32,130 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:32,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:32,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:32,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:32,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:32,144 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:32,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@143f31b9[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:32,145 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:32,145 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:32,145 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:32,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:32,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:32,146 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:32,146 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:32,146 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:32,146 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,146 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:32,147 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,147 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:32,147 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:32,147 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:32,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:32,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:32,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:32,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:32,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:32,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:32,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:32,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on c79c35ba58052cacab6709e8682a60f1 2023-03-20 10:56:32,151 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c79c35ba58052cacab6709e8682a60f1 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-20 10:56:32,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,152 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=A 2023-03-20 10:56:32,152 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:32,152 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=B 2023-03-20 10:56:32,152 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:32,152 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=C 2023-03-20 10:56:32,152 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:32,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:32,153 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:32,153 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:32,153 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:32,153 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:32,153 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:32,153 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:32,160 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:32,160 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:32,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:32,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:32,161 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:32,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:32,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:32,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:32,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:32,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:32,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:32,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:32,164 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:32,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:32,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:32,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,167 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:32,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:32,169 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:32,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:32,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:32,169 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:32,169 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,169 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:32,169 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:32,169 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:32,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:32,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:32,169 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,170 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:32,170 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:32,170 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:32,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:32,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:32,170 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:32,170 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:32,170 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:56:32,197 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=399 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/A/a4a5c5e3266e41e5ba27e45d9dcf6bbf 2023-03-20 10:56:32,215 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=399 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/B/8a503bcb43d747a5b70c610fd865e1b6 2023-03-20 10:56:32,249 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679309792201, max timestamp=1679309792246Num uniques 859; 2023-03-20 10:56:32,249 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-20 10:56:32,250 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679309792201, max timestamp=1679309792246Num uniques 859; 2023-03-20 10:56:32,250 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-20 10:56:32,252 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679309792201, max timestamp=1679309792246Num uniques 859; 2023-03-20 10:56:32,252 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-20 10:56:32,259 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=399 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/C/3b338e20b5504ab1bf74a771fa854ee7 2023-03-20 10:56:32,266 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/A/a4a5c5e3266e41e5ba27e45d9dcf6bbf as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/a4a5c5e3266e41e5ba27e45d9dcf6bbf 2023-03-20 10:56:32,270 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:32,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:32,274 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/a4a5c5e3266e41e5ba27e45d9dcf6bbf, entries=200, sequenceid=399, filesize=14.3 K 2023-03-20 10:56:32,276 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/B/8a503bcb43d747a5b70c610fd865e1b6 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/8a503bcb43d747a5b70c610fd865e1b6 2023-03-20 10:56:32,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:32,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:32,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:32,278 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:32,279 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:32,279 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:32,279 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:32,279 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:32,279 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:32,280 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:32,280 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:32,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:32,281 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,281 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:32,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:32,282 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:32,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:32,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:32,283 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:32,285 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/8a503bcb43d747a5b70c610fd865e1b6, entries=150, sequenceid=399, filesize=11.9 K 2023-03-20 10:56:32,286 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:32,286 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:32,286 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:32,286 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:32,286 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:32,287 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/C/3b338e20b5504ab1bf74a771fa854ee7 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/3b338e20b5504ab1bf74a771fa854ee7 2023-03-20 10:56:32,304 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:32,304 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:32,304 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:32,304 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. started... 2023-03-20 10:56:32,304 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. as already flushing 2023-03-20 10:56:32,304 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:32,304 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:32,304 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:32,304 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,305 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:32,305 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:32,306 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/3b338e20b5504ab1bf74a771fa854ee7, entries=150, sequenceid=399, filesize=11.9 K 2023-03-20 10:56:32,307 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:32,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:32,308 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:32,308 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=87.22 KB/89310 for c79c35ba58052cacab6709e8682a60f1 in 156ms, sequenceid=399, compaction requested=true 2023-03-20 10:56:32,308 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c79c35ba58052cacab6709e8682a60f1: 2023-03-20 10:56:32,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:32,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:32,309 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:32,309 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:32,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:32,309 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 10:56:32,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,309 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:56:32,309 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 10:56:32,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:32,309 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:56:32,311 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 54136 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 10:56:32,311 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1912): c79c35ba58052cacab6709e8682a60f1/A is initiating minor compaction (all files) 2023-03-20 10:56:32,311 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c79c35ba58052cacab6709e8682a60f1/A in TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:32,311 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/19e5b29188d243bd9aec3d472c4d1131, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/c709adc09d36496e941d6f522cc07b84, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/76794a01628e4aa5b96ad3cc301cbe32, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/a4a5c5e3266e41e5ba27e45d9dcf6bbf] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp, totalSize=52.9 K 2023-03-20 10:56:32,312 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:56:32,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:32,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d09fb87[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:32,312 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:32,312 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:32,313 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49256 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 10:56:32,313 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): c79c35ba58052cacab6709e8682a60f1/B is initiating minor compaction (all files) 2023-03-20 10:56:32,313 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c79c35ba58052cacab6709e8682a60f1/B in TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:32,313 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/cdeedb8f493848b0a6d4169dfda1f961, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/ff7e43113ce64683a932a0811942de17, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/682afc884e684f7c8794383ebb8cdbb1, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/8a503bcb43d747a5b70c610fd865e1b6] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp, totalSize=48.1 K 2023-03-20 10:56:32,313 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:32,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:32,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:32,314 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:32,314 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:32,314 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:32,314 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 19e5b29188d243bd9aec3d472c4d1131, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=353, earliestPutTs=1679309790544 2023-03-20 10:56:32,314 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,314 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:32,316 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting cdeedb8f493848b0a6d4169dfda1f961, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=353, earliestPutTs=1679309790544 2023-03-20 10:56:32,317 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting ff7e43113ce64683a932a0811942de17, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=368, earliestPutTs=1679309790701 2023-03-20 10:56:32,317 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting c709adc09d36496e941d6f522cc07b84, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=368, earliestPutTs=1679309790701 2023-03-20 10:56:32,318 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 682afc884e684f7c8794383ebb8cdbb1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=389, earliestPutTs=1679309791386 2023-03-20 10:56:32,318 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:32,318 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:32,318 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:32,318 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:32,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:32,319 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 76794a01628e4aa5b96ad3cc301cbe32, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=389, earliestPutTs=1679309791386 2023-03-20 10:56:32,320 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 8a503bcb43d747a5b70c610fd865e1b6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=399, earliestPutTs=1679309792113 2023-03-20 10:56:32,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:32,320 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting a4a5c5e3266e41e5ba27e45d9dcf6bbf, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=399, earliestPutTs=1679309792113 2023-03-20 10:56:32,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:32,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:32,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on c79c35ba58052cacab6709e8682a60f1 2023-03-20 10:56:32,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:32,326 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c79c35ba58052cacab6709e8682a60f1 3/3 column families, dataSize=93.93 KB heapSize=260.77 KB 2023-03-20 10:56:32,327 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=A 2023-03-20 10:56:32,327 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:56:32,327 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=B 2023-03-20 10:56:32,327 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:56:32,327 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=C 2023-03-20 10:56:32,327 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:56:32,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:32,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:32,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:32,336 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:32,336 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:32,336 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:32,336 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:32,336 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:32,339 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:32,339 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:32,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:32,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:32,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:32,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:32,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:32,339 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:32,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:32,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:32,340 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:32,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:32,344 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:32,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:32,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:32,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,356 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:32,357 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:32,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:32,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:32,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:32,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:32,360 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] throttle.PressureAwareThroughputController(145): c79c35ba58052cacab6709e8682a60f1#A#compaction#58 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:56:32,364 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:32,364 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:32,364 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): c79c35ba58052cacab6709e8682a60f1#B#compaction#59 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:56:32,364 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:32,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:32,365 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,365 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:32,365 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:32,365 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,365 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:32,365 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:32,365 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:32,366 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:32,366 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:32,371 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:56:32,464 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:32,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41456 deadline: 1679309852463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,466 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:32,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:32,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:32,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:32,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:32,502 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:32,502 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:32,502 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:32,502 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:32,504 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:32,504 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:32,505 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,505 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:32,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:32,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:32,506 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,506 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:32,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:32,506 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:32,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:32,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:32,507 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:32,526 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:32,526 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:32,526 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:32,526 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:32,526 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:32,537 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:32,537 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:32,539 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:32,539 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. started... 2023-03-20 10:56:32,539 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. as already flushing 2023-03-20 10:56:32,540 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:32,540 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:32,540 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:32,540 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,540 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:32,540 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:32,541 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:32,541 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:32,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:32,542 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:32,542 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:32,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:32,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:32,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:32,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:32,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:32,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b39c98c[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:32,546 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:32,546 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:32,547 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:32,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:32,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:32,547 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:32,548 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:32,548 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:32,548 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,548 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:32,549 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,549 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:32,549 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:32,549 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:32,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:32,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:32,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:32,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:32,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:32,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:32,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:32,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:32,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:32,552 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:32,552 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:32,553 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:32,553 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:32,553 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:32,562 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:32,562 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:32,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:32,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:32,563 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:32,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:32,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:32,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:32,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:32,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:32,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:32,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:32,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:32,565 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:32,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:32,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:32,567 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:32,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,569 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:32,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:32,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:32,570 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:32,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:32,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:32,570 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,570 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:32,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:32,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:32,570 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:32,571 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:32,571 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,571 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:32,571 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:32,571 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:32,571 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:32,571 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:32,576 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:32,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41456 deadline: 1679309852566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,671 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:32,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:32,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:32,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:32,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:32,688 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:32,688 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:32,688 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:32,688 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:32,689 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:32,689 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:32,691 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:32,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:32,691 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:32,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:32,692 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,692 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:32,692 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:32,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:32,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:32,694 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:32,694 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:32,694 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:32,694 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:32,694 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:32,694 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:32,697 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:32,697 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:32,697 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:32,697 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. started... 2023-03-20 10:56:32,698 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. as already flushing 2023-03-20 10:56:32,698 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:32,698 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:32,698 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:32,698 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,698 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:32,698 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:32,703 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:32,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:32,703 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:32,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:32,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:32,704 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:32,704 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:32,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:32,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:32,719 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:32,719 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6457aa34[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:32,719 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:32,719 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:32,721 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,721 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:32,722 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:32,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:32,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:32,722 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:32,722 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:32,722 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:32,723 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:32,723 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,723 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:32,723 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:32,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:32,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:32,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:32,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:32,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:32,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:32,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:32,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:32,726 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:32,726 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:32,726 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:32,726 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:32,726 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:32,726 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:32,728 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:32,728 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:32,728 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:32,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:32,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:32,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:32,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:32,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:32,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:32,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:32,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:32,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:32,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:32,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:32,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:32,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:32,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:32,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,738 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,771 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:32,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41436 deadline: 1679309852770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,773 DEBUG [Thread-145] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8200 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1., hostname=jenkins-hbase20.apache.org,36041,1679309777062, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-20 10:56:32,774 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,776 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,776 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:32,777 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:32,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:32,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:32,776 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:32,777 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:32,777 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,777 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:32,777 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:32,777 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:32,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:32,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:32,777 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:32,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:32,778 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:32,778 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:32,778 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:32,791 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:32,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41456 deadline: 1679309852785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,815 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:32,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41442 deadline: 1679309852811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,820 DEBUG [Thread-141] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8244 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1., hostname=jenkins-hbase20.apache.org,36041,1679309777062, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-20 10:56:32,836 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:32,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41388 deadline: 1679309852831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,837 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:32,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:41416 deadline: 1679309852828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,839 DEBUG [Thread-147] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8265 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1., hostname=jenkins-hbase20.apache.org,36041,1679309777062, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-20 10:56:32,842 DEBUG [Thread-143] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8261 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1., hostname=jenkins-hbase20.apache.org,36041,1679309777062, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-20 10:56:32,853 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=429 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/A/abb3833bf0d04557a5bee373c8632798 2023-03-20 10:56:32,876 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/B/3a3ce55963f24bb78b7ec45526c2d082 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/3a3ce55963f24bb78b7ec45526c2d082 2023-03-20 10:56:32,878 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:32,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:32,881 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/A/5f913c4c34a54f6e9b8aca5e554e9ca1 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/5f913c4c34a54f6e9b8aca5e554e9ca1 2023-03-20 10:56:32,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:32,893 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c79c35ba58052cacab6709e8682a60f1/B of c79c35ba58052cacab6709e8682a60f1 into 3a3ce55963f24bb78b7ec45526c2d082(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:56:32,893 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c79c35ba58052cacab6709e8682a60f1: 2023-03-20 10:56:32,893 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1., storeName=c79c35ba58052cacab6709e8682a60f1/B, priority=12, startTime=1679309792309; duration=0sec 2023-03-20 10:56:32,893 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:56:32,893 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 10:56:32,894 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c79c35ba58052cacab6709e8682a60f1/A of c79c35ba58052cacab6709e8682a60f1 into 5f913c4c34a54f6e9b8aca5e554e9ca1(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:56:32,894 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c79c35ba58052cacab6709e8682a60f1: 2023-03-20 10:56:32,894 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1., storeName=c79c35ba58052cacab6709e8682a60f1/A, priority=12, startTime=1679309792309; duration=0sec 2023-03-20 10:56:32,894 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:56:32,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:32,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:32,897 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49256 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 10:56:32,898 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): c79c35ba58052cacab6709e8682a60f1/C is initiating minor compaction (all files) 2023-03-20 10:56:32,898 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c79c35ba58052cacab6709e8682a60f1/C in TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:32,898 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/12919e9f7e1e4b91904908a980b56be4, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/977f0a046b504392896f1de49b3d7d6f, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/e9b8cc68bbc1490bb6f47317a6e320a3, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/3b338e20b5504ab1bf74a771fa854ee7] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp, totalSize=48.1 K 2023-03-20 10:56:32,899 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 12919e9f7e1e4b91904908a980b56be4, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=353, earliestPutTs=1679309790544 2023-03-20 10:56:32,899 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:32,899 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:32,899 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:32,899 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:32,899 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 977f0a046b504392896f1de49b3d7d6f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=368, earliestPutTs=1679309790701 2023-03-20 10:56:32,900 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting e9b8cc68bbc1490bb6f47317a6e320a3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=389, earliestPutTs=1679309791386 2023-03-20 10:56:32,900 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 3b338e20b5504ab1bf74a771fa854ee7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=399, earliestPutTs=1679309792113 2023-03-20 10:56:32,901 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:32,901 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:32,902 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=429 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/B/51cf546658c74aba849b6aece7774cf7 2023-03-20 10:56:32,902 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:32,902 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:32,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:32,903 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,903 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:32,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:32,904 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:32,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:32,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:32,904 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:32,904 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:32,904 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:32,904 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:32,904 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:32,914 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:32,914 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:32,914 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:32,914 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. started... 2023-03-20 10:56:32,914 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. as already flushing 2023-03-20 10:56:32,914 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:32,914 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:32,914 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:32,915 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,924 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:32,924 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:32,925 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:32,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:32,925 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:32,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:32,925 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): c79c35ba58052cacab6709e8682a60f1#C#compaction#62 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:56:32,926 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:32,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:32,926 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:32,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:32,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:32,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:32,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@988edfe[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:32,930 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:32,930 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:32,936 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:32,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:32,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:32,936 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:32,936 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:32,936 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:32,936 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,936 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:32,937 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:32,937 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,937 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:32,937 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:32,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:32,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:32,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:32,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:32,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:32,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:32,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:32,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:32,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:32,946 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:32,946 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:32,946 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:32,946 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:32,946 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:32,952 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:32,952 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:32,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:32,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:32,953 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:32,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:32,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:32,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:32,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:32,953 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:32,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:32,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:32,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:32,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:32,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:32,959 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:32,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:32,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,968 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=429 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/C/5752a8ac55a14723bb95453c29d4139e 2023-03-20 10:56:32,969 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:32,969 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:32,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:32,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:32,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:32,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:32,970 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,970 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:32,970 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:32,970 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:32,970 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:32,971 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:32,971 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:32,970 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:32,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:32,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:32,975 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/C/d4f2e38940864b24bf3c1cc838570cb7 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/d4f2e38940864b24bf3c1cc838570cb7 2023-03-20 10:56:32,979 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:56:32,981 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:32,982 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:32,982 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/A/abb3833bf0d04557a5bee373c8632798 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/abb3833bf0d04557a5bee373c8632798 2023-03-20 10:56:32,990 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c79c35ba58052cacab6709e8682a60f1/C of c79c35ba58052cacab6709e8682a60f1 into d4f2e38940864b24bf3c1cc838570cb7(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:56:32,990 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c79c35ba58052cacab6709e8682a60f1: 2023-03-20 10:56:32,990 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1., storeName=c79c35ba58052cacab6709e8682a60f1/C, priority=12, startTime=1679309792312; duration=0sec 2023-03-20 10:56:32,990 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:56:32,991 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/abb3833bf0d04557a5bee373c8632798, entries=200, sequenceid=429, filesize=14.3 K 2023-03-20 10:56:32,993 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/B/51cf546658c74aba849b6aece7774cf7 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/51cf546658c74aba849b6aece7774cf7 2023-03-20 10:56:33,002 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/51cf546658c74aba849b6aece7774cf7, entries=150, sequenceid=429, filesize=11.9 K 2023-03-20 10:56:33,004 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/C/5752a8ac55a14723bb95453c29d4139e as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/5752a8ac55a14723bb95453c29d4139e 2023-03-20 10:56:33,013 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/5752a8ac55a14723bb95453c29d4139e, entries=150, sequenceid=429, filesize=11.9 K 2023-03-20 10:56:33,014 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~260.72 KB/266976, currentSize=87.22 KB/89310 for c79c35ba58052cacab6709e8682a60f1 in 688ms, sequenceid=429, compaction requested=false 2023-03-20 10:56:33,014 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c79c35ba58052cacab6709e8682a60f1: 2023-03-20 10:56:33,082 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:33,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:33,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:33,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:33,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:33,088 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:33,088 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:33,088 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:33,088 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:33,088 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:33,088 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:33,089 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:33,089 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:33,089 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:33,090 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:33,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:33,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:33,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:33,090 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:33,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:33,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:33,091 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:33,091 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:33,091 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:33,091 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:33,091 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:33,091 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:33,091 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:33,091 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:33,091 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:33,091 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. started... 2023-03-20 10:56:33,092 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c79c35ba58052cacab6709e8682a60f1 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-20 10:56:33,092 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=A 2023-03-20 10:56:33,092 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:33,092 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=B 2023-03-20 10:56:33,092 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:33,092 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=C 2023-03-20 10:56:33,092 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:33,096 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. as already flushing 2023-03-20 10:56:33,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on c79c35ba58052cacab6709e8682a60f1 2023-03-20 10:56:33,151 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=448 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/A/c7c518a9f4124023aadf3a7e2a39e3ca 2023-03-20 10:56:33,177 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:33,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41456 deadline: 1679309853174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:33,203 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=448 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/B/ae791a0044fb4bb99372369d5005a959 2023-03-20 10:56:33,282 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:33,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 294 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41456 deadline: 1679309853278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:33,495 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:33,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 296 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41456 deadline: 1679309853495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:33,656 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=448 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/C/9115a29c7faf4f7390d6fec9dbb2167b 2023-03-20 10:56:33,667 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/A/c7c518a9f4124023aadf3a7e2a39e3ca as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/c7c518a9f4124023aadf3a7e2a39e3ca 2023-03-20 10:56:33,707 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/c7c518a9f4124023aadf3a7e2a39e3ca, entries=150, sequenceid=448, filesize=11.9 K 2023-03-20 10:56:33,711 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/B/ae791a0044fb4bb99372369d5005a959 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/ae791a0044fb4bb99372369d5005a959 2023-03-20 10:56:33,722 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/ae791a0044fb4bb99372369d5005a959, entries=150, sequenceid=448, filesize=11.9 K 2023-03-20 10:56:33,727 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/C/9115a29c7faf4f7390d6fec9dbb2167b as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/9115a29c7faf4f7390d6fec9dbb2167b 2023-03-20 10:56:33,752 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/9115a29c7faf4f7390d6fec9dbb2167b, entries=150, sequenceid=448, filesize=11.9 K 2023-03-20 10:56:33,767 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for c79c35ba58052cacab6709e8682a60f1 in 675ms, sequenceid=448, compaction requested=true 2023-03-20 10:56:33,767 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c79c35ba58052cacab6709e8682a60f1: 2023-03-20 10:56:33,767 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:33,767 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:33,767 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:33,767 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:33,767 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:33,767 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:33,769 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:33,770 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:33,770 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:33,772 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:33,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:33,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:33,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:33,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:33,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:33,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:33,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:33,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:33,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:33,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b9ffa8b[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:33,800 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:33,800 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:33,803 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:33,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:33,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:33,805 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:33,806 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:33,806 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:33,806 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:33,806 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:33,812 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:33,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:33,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:33,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:33,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on c79c35ba58052cacab6709e8682a60f1 2023-03-20 10:56:33,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:33,813 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c79c35ba58052cacab6709e8682a60f1 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-20 10:56:33,813 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=A 2023-03-20 10:56:33,813 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:33,813 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=B 2023-03-20 10:56:33,813 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:33,813 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=C 2023-03-20 10:56:33,813 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:33,814 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:33,814 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:33,814 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:33,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:33,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:33,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:33,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:33,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:33,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:33,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:33,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:33,821 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:33,821 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:33,821 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:33,821 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:33,821 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:33,823 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:33,824 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:33,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:33,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:33,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:33,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:33,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:33,825 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:33,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:33,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:33,826 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:33,826 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:33,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:33,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:33,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:33,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:33,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:33,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:33,831 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:33,831 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:33,831 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:33,831 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:33,831 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:33,831 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:33,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:33,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:33,831 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:56:33,831 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:33,832 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:33,832 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:33,832 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:33,834 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:33,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:33,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:33,932 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:33,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:33,936 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:33,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:33,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 311 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41456 deadline: 1679309853936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:33,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:33,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:33,951 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:33,951 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:33,951 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:33,951 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:33,951 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:33,951 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:33,952 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:33,952 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:33,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:33,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:33,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:33,953 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:33,953 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:33,953 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:33,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:33,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:33,954 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:33,954 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:33,955 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:33,955 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:33,955 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:33,955 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:33,955 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:33,955 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:33,955 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:33,955 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. started... 2023-03-20 10:56:33,955 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. as already flushing 2023-03-20 10:56:33,955 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:33,955 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:33,955 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:33,955 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:33,955 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:33,955 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:33,957 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:33,957 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:33,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:33,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:33,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:33,957 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:33,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:33,957 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:33,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:33,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:33,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:33,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:33,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:33,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e3f3fdd[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:33,971 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:33,971 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:33,977 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:33,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:33,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:33,977 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:33,977 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:33,977 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:33,977 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:33,978 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:33,985 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:33,986 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:33,986 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:33,986 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:33,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:33,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:33,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:33,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:33,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:33,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:33,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:33,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:33,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:33,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:33,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:33,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:33,993 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:33,993 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:33,993 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:33,993 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:33,993 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:33,995 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:33,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:33,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:33,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:33,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:33,996 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:33,996 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:33,996 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:33,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:33,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:33,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:33,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:33,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:33,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:33,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:33,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:33,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:33,999 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:33,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:33,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:34,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,001 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:34,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:34,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:34,002 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:34,002 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:34,002 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,002 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:34,002 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:34,002 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,002 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:34,002 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:34,002 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:34,002 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:34,002 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:34,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:34,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:34,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:34,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:34,041 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:34,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 313 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41456 deadline: 1679309854039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,102 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:34,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:34,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:34,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:34,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:34,119 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:34,119 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:34,119 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:34,119 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:34,119 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:34,119 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:34,120 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,120 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:34,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:34,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:34,121 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,121 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:34,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:34,122 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:34,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:34,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:34,124 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:34,130 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:34,130 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:34,130 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:34,130 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:34,130 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:34,154 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:34,154 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:34,156 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:34,156 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. started... 2023-03-20 10:56:34,156 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. as already flushing 2023-03-20 10:56:34,156 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:34,156 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:34,156 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:34,156 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,156 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:34,156 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:34,158 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,158 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:34,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:34,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:34,158 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:34,158 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:34,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:34,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:34,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:34,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:34,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:34,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47e03024[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:34,162 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:34,162 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:34,163 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:34,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:34,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:34,163 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:34,163 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:34,163 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:34,163 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,163 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:34,164 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:34,164 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:34,164 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:34,164 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:34,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:34,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:34,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:34,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:34,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:34,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:34,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:34,167 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:34,167 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:34,167 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:34,167 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:34,167 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:34,167 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:34,169 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:34,169 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:34,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:34,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:34,169 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:34,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:34,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:34,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:34,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:34,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:34,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:34,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:34,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:34,170 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:34,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:34,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,171 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:34,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:34,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,173 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:34,173 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:34,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:34,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:34,173 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:34,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:34,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:34,173 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:34,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:34,174 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:34,174 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:34,173 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:34,174 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:34,174 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,174 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:34,174 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:34,174 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:34,247 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:34,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 315 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41456 deadline: 1679309854244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,255 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=465 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/A/c2f5a9522cc745d98a47d7ba697b77a9 2023-03-20 10:56:34,274 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:34,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:34,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:34,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:34,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:34,298 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:34,298 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:34,298 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:34,298 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:34,299 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:34,299 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:34,300 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,300 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:34,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:34,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:34,301 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,301 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:34,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:34,301 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:34,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:34,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:34,301 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:34,318 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:34,318 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:34,318 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:34,319 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:34,319 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:34,322 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:34,322 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:34,322 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:34,322 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. started... 2023-03-20 10:56:34,323 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. as already flushing 2023-03-20 10:56:34,323 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:34,323 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:34,323 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:34,323 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,323 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:34,323 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:34,328 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:34,328 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:34,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:34,329 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:34,329 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:34,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:34,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:34,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:34,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:34,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:34,332 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@949d23b[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:34,332 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:34,332 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:34,335 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:34,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:34,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:34,336 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:34,336 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:34,336 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,336 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:34,336 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:34,336 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=465 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/B/c6ff30ad3b8b44dcbc17365c40828c44 2023-03-20 10:56:34,344 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:34,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:34,344 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:34,344 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:34,344 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:34,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:34,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:34,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:34,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:34,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:34,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:34,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:34,350 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:34,350 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:34,350 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:34,350 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:34,350 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:34,352 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:34,352 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:34,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:34,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:34,353 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:34,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:34,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:34,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:34,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:34,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:34,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:34,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:34,354 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:34,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:34,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:34,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,356 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:34,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:34,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,361 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:34,361 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:34,361 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,362 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:34,362 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:34,362 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,362 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:34,362 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:34,362 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:34,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:34,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:34,362 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:34,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:34,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:34,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:34,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:34,363 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:34,363 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:34,463 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:34,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:34,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:34,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:34,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:34,475 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:34,475 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:34,475 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:34,475 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:34,476 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:34,476 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:34,477 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:34,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:34,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:34,478 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:34,478 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:34,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:34,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:34,479 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:34,479 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,479 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:34,483 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:34,483 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:34,483 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:34,483 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:34,483 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:34,487 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:34,487 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:34,497 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:34,497 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. started... 2023-03-20 10:56:34,497 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. as already flushing 2023-03-20 10:56:34,497 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:34,497 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:34,497 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:34,497 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,497 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:34,497 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:34,499 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:34,499 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,500 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:34,500 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:34,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:34,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:34,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:34,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:34,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:34,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:34,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:34,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d362a3b[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:34,504 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:34,504 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:34,511 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:34,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:34,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:34,511 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:34,511 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:34,511 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:34,512 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,512 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:34,518 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:34,518 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:34,519 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:34,520 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:34,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:34,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:34,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:34,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:34,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:34,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:34,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:34,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:34,526 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:34,526 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:34,526 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:34,526 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:34,526 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:34,528 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:34,528 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:34,528 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:34,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:34,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:34,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:34,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:34,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:34,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:34,529 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:34,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:34,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:34,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:34,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:34,532 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:34,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:34,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:34,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,534 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:34,534 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:34,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:34,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:34,535 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,535 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:34,535 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:34,535 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,535 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:34,535 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:34,535 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:34,535 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:34,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:34,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:34,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:34,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:34,535 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:56:34,535 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:34,535 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:34,552 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:34,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 317 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41456 deadline: 1679309854552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,636 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:34,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:34,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:34,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:34,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:34,651 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:34,651 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:34,651 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:34,651 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:34,651 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:34,651 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:34,652 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,653 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:34,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:34,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:34,653 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,653 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:34,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:34,653 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:34,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:34,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:34,654 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:34,667 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:34,667 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:34,667 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:34,667 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:34,667 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:34,689 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:34,689 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:34,694 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:34,694 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. started... 2023-03-20 10:56:34,695 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. as already flushing 2023-03-20 10:56:34,695 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:34,695 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:34,695 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:34,695 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,695 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:34,695 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:34,699 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,700 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:34,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:34,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:34,701 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:34,701 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:34,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:34,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:34,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:34,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:34,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:34,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a102b97[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:34,703 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:34,703 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:34,704 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:34,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:34,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:34,704 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:34,704 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:34,704 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:34,704 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,704 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:34,706 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:34,706 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:34,706 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:34,707 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:34,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:34,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:34,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:34,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:34,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:34,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:34,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:34,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:34,729 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:34,730 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:34,730 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:34,730 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:34,730 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:34,745 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:34,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:34,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:34,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:34,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:34,746 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:34,746 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:34,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:34,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:34,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:34,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:34,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:34,748 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:34,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:34,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:34,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:34,749 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:34,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,759 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:34,759 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:34,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:34,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:34,759 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,759 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:34,759 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:34,760 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,760 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:34,760 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:34,760 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:34,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:34,760 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:34,760 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:34,760 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:34,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:34,762 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:34,762 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:34,834 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=465 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/C/d4c5b134217f4b00b949766cce010ecd 2023-03-20 10:56:34,844 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/A/c2f5a9522cc745d98a47d7ba697b77a9 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/c2f5a9522cc745d98a47d7ba697b77a9 2023-03-20 10:56:34,853 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/c2f5a9522cc745d98a47d7ba697b77a9, entries=200, sequenceid=465, filesize=14.3 K 2023-03-20 10:56:34,856 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/B/c6ff30ad3b8b44dcbc17365c40828c44 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/c6ff30ad3b8b44dcbc17365c40828c44 2023-03-20 10:56:34,862 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:34,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:34,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:34,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:34,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:34,878 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:34,878 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:34,878 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:34,878 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:34,879 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:34,879 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:34,880 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:34,880 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:34,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:34,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:34,881 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:34,881 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,881 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:34,881 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/c6ff30ad3b8b44dcbc17365c40828c44, entries=150, sequenceid=465, filesize=11.9 K 2023-03-20 10:56:34,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:34,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:34,883 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:34,883 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:34,883 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:34,883 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:34,883 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:34,883 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:34,884 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/C/d4c5b134217f4b00b949766cce010ecd as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/d4c5b134217f4b00b949766cce010ecd 2023-03-20 10:56:34,887 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:34,887 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:34,894 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:34,894 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. started... 2023-03-20 10:56:34,894 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. as already flushing 2023-03-20 10:56:34,894 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:34,894 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:34,894 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:34,894 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,896 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:34,896 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:34,898 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:34,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:34,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:34,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:34,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:34,899 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:34,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:34,900 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:34,900 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:34,900 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:34,900 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1cb751e1[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:34,900 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:34,900 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:34,901 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:34,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:34,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:34,901 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:34,901 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:34,901 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:34,901 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,901 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:34,902 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:34,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:34,902 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:34,902 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:34,902 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:34,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:34,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:34,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:34,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:34,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:34,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:34,905 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/d4c5b134217f4b00b949766cce010ecd, entries=150, sequenceid=465, filesize=11.9 K 2023-03-20 10:56:34,905 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:34,905 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:34,905 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:34,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:34,906 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:34,906 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:34,907 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for c79c35ba58052cacab6709e8682a60f1 in 1094ms, sequenceid=465, compaction requested=true 2023-03-20 10:56:34,907 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c79c35ba58052cacab6709e8682a60f1: 2023-03-20 10:56:34,907 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:56:34,907 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 10:56:34,907 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:56:34,907 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-20 10:56:34,908 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:34,908 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:34,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:34,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:34,908 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 10:56:34,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:34,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:34,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:34,912 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49392 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 10:56:34,912 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): c79c35ba58052cacab6709e8682a60f1/B is initiating minor compaction (all files) 2023-03-20 10:56:34,912 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c79c35ba58052cacab6709e8682a60f1/B in TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:34,912 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/3a3ce55963f24bb78b7ec45526c2d082, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/51cf546658c74aba849b6aece7774cf7, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/ae791a0044fb4bb99372369d5005a959, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/c6ff30ad3b8b44dcbc17365c40828c44] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp, totalSize=48.2 K 2023-03-20 10:56:34,913 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 3a3ce55963f24bb78b7ec45526c2d082, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=399, earliestPutTs=1679309792113 2023-03-20 10:56:34,914 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:34,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:34,914 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 51cf546658c74aba849b6aece7774cf7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=429, earliestPutTs=1679309792291 2023-03-20 10:56:34,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:34,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:34,915 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting ae791a0044fb4bb99372369d5005a959, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=448, earliestPutTs=1679309792434 2023-03-20 10:56:34,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:34,915 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting c6ff30ad3b8b44dcbc17365c40828c44, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=465, earliestPutTs=1679309793134 2023-03-20 10:56:34,915 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:34,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:34,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:34,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,918 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:34,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:34,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,922 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:34,922 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:56:34,922 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:34,922 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:34,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:34,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:34,922 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:34,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:34,922 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:34,922 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:34,922 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:34,923 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:34,923 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:34,923 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:34,923 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:34,923 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:34,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:34,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:34,928 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 54272 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 10:56:34,928 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1912): c79c35ba58052cacab6709e8682a60f1/A is initiating minor compaction (all files) 2023-03-20 10:56:34,928 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c79c35ba58052cacab6709e8682a60f1/A in TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:34,928 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/5f913c4c34a54f6e9b8aca5e554e9ca1, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/abb3833bf0d04557a5bee373c8632798, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/c7c518a9f4124023aadf3a7e2a39e3ca, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/c2f5a9522cc745d98a47d7ba697b77a9] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp, totalSize=53 K 2023-03-20 10:56:34,929 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 5f913c4c34a54f6e9b8aca5e554e9ca1, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=399, earliestPutTs=1679309792113 2023-03-20 10:56:34,930 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting abb3833bf0d04557a5bee373c8632798, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=429, earliestPutTs=1679309792288 2023-03-20 10:56:34,931 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting c7c518a9f4124023aadf3a7e2a39e3ca, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=448, earliestPutTs=1679309792434 2023-03-20 10:56:34,931 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting c2f5a9522cc745d98a47d7ba697b77a9, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=465, earliestPutTs=1679309793134 2023-03-20 10:56:34,952 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): c79c35ba58052cacab6709e8682a60f1#B#compaction#69 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:56:34,956 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] throttle.PressureAwareThroughputController(145): c79c35ba58052cacab6709e8682a60f1#A#compaction#70 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:56:35,003 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/B/2b0dde7e4974450c99f04e9867a2ffd3 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/2b0dde7e4974450c99f04e9867a2ffd3 2023-03-20 10:56:35,015 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c79c35ba58052cacab6709e8682a60f1/B of c79c35ba58052cacab6709e8682a60f1 into 2b0dde7e4974450c99f04e9867a2ffd3(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:56:35,015 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c79c35ba58052cacab6709e8682a60f1: 2023-03-20 10:56:35,015 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1., storeName=c79c35ba58052cacab6709e8682a60f1/B, priority=12, startTime=1679309794907; duration=0sec 2023-03-20 10:56:35,015 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:56:35,015 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 10:56:35,019 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49392 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 10:56:35,019 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): c79c35ba58052cacab6709e8682a60f1/C is initiating minor compaction (all files) 2023-03-20 10:56:35,019 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c79c35ba58052cacab6709e8682a60f1/C in TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:35,019 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/d4f2e38940864b24bf3c1cc838570cb7, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/5752a8ac55a14723bb95453c29d4139e, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/9115a29c7faf4f7390d6fec9dbb2167b, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/d4c5b134217f4b00b949766cce010ecd] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp, totalSize=48.2 K 2023-03-20 10:56:35,021 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting d4f2e38940864b24bf3c1cc838570cb7, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=399, earliestPutTs=1679309792113 2023-03-20 10:56:35,022 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 5752a8ac55a14723bb95453c29d4139e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=429, earliestPutTs=1679309792291 2023-03-20 10:56:35,023 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:35,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:35,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:35,027 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 9115a29c7faf4f7390d6fec9dbb2167b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=448, earliestPutTs=1679309792434 2023-03-20 10:56:35,031 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting d4c5b134217f4b00b949766cce010ecd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=465, earliestPutTs=1679309793134 2023-03-20 10:56:35,032 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/A/942c1a5846f14fcc9bc9f5cd62570a29 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/942c1a5846f14fcc9bc9f5cd62570a29 2023-03-20 10:56:35,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:35,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:35,034 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:35,034 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:35,034 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:35,034 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:35,035 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:35,035 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:35,036 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:35,036 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:35,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:35,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:35,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:35,036 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:35,036 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:35,037 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:35,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:35,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:35,038 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:35,041 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:35,041 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:35,041 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:35,041 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:35,041 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:35,046 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:35,046 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:35,046 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:35,046 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. started... 2023-03-20 10:56:35,046 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c79c35ba58052cacab6709e8682a60f1 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-20 10:56:35,048 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c79c35ba58052cacab6709e8682a60f1/A of c79c35ba58052cacab6709e8682a60f1 into 942c1a5846f14fcc9bc9f5cd62570a29(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:56:35,048 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c79c35ba58052cacab6709e8682a60f1: 2023-03-20 10:56:35,049 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1., storeName=c79c35ba58052cacab6709e8682a60f1/A, priority=12, startTime=1679309794907; duration=0sec 2023-03-20 10:56:35,049 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:56:35,049 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=A 2023-03-20 10:56:35,050 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:35,050 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=B 2023-03-20 10:56:35,050 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:35,050 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=C 2023-03-20 10:56:35,050 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:35,053 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): c79c35ba58052cacab6709e8682a60f1#C#compaction#71 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:56:35,068 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. as already flushing 2023-03-20 10:56:35,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on c79c35ba58052cacab6709e8682a60f1 2023-03-20 10:56:35,135 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:35,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 333 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41456 deadline: 1679309855135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:35,237 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:35,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 335 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41456 deadline: 1679309855237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:35,440 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:35,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 337 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41456 deadline: 1679309855439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:35,523 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=482 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/A/cd3b7cc9b2a24779810b424c3bf72a37 2023-03-20 10:56:35,555 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/C/a1181ab928014935aafea6b32f9cf36b as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/a1181ab928014935aafea6b32f9cf36b 2023-03-20 10:56:35,576 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c79c35ba58052cacab6709e8682a60f1/C of c79c35ba58052cacab6709e8682a60f1 into a1181ab928014935aafea6b32f9cf36b(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:56:35,576 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c79c35ba58052cacab6709e8682a60f1: 2023-03-20 10:56:35,576 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1., storeName=c79c35ba58052cacab6709e8682a60f1/C, priority=12, startTime=1679309794907; duration=0sec 2023-03-20 10:56:35,576 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:56:35,744 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:35,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 339 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41456 deadline: 1679309855744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:36,038 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=482 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/B/5838a70c16524ac49b30fd9a085bde64 2023-03-20 10:56:36,132 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=482 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/C/b1ade1a10b5e4669bf491d676c5cffb4 2023-03-20 10:56:36,140 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/A/cd3b7cc9b2a24779810b424c3bf72a37 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/cd3b7cc9b2a24779810b424c3bf72a37 2023-03-20 10:56:36,147 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/cd3b7cc9b2a24779810b424c3bf72a37, entries=150, sequenceid=482, filesize=11.9 K 2023-03-20 10:56:36,149 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/B/5838a70c16524ac49b30fd9a085bde64 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/5838a70c16524ac49b30fd9a085bde64 2023-03-20 10:56:36,174 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/5838a70c16524ac49b30fd9a085bde64, entries=150, sequenceid=482, filesize=11.9 K 2023-03-20 10:56:36,182 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/C/b1ade1a10b5e4669bf491d676c5cffb4 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/b1ade1a10b5e4669bf491d676c5cffb4 2023-03-20 10:56:36,210 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/b1ade1a10b5e4669bf491d676c5cffb4, entries=150, sequenceid=482, filesize=11.9 K 2023-03-20 10:56:36,218 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for c79c35ba58052cacab6709e8682a60f1 in 1172ms, sequenceid=482, compaction requested=false 2023-03-20 10:56:36,218 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c79c35ba58052cacab6709e8682a60f1: 2023-03-20 10:56:36,218 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:36,218 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:36,218 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:36,218 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:36,218 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:36,218 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:36,221 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:36,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:36,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:36,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:36,221 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:36,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:36,222 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:36,222 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:36,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:36,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:36,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:36,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:36,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:36,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ba89046[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:36,234 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:36,235 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:36,254 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:36,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:36,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:36,255 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:36,255 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:36,255 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:36,255 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:36,255 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:36,256 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:36,256 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:36,256 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:36,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:36,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:36,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:36,256 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:36,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:36,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:36,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:36,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:36,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on c79c35ba58052cacab6709e8682a60f1 2023-03-20 10:56:36,291 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c79c35ba58052cacab6709e8682a60f1 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-20 10:56:36,291 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=A 2023-03-20 10:56:36,292 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:36,292 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=B 2023-03-20 10:56:36,292 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:36,292 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=C 2023-03-20 10:56:36,292 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:36,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:36,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:36,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:36,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:36,297 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:36,297 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:36,297 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:36,297 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:36,297 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:36,297 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:36,301 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:36,301 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:36,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:36,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:36,301 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:36,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:36,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:36,301 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:36,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:36,303 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:36,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:36,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:36,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:36,305 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:36,305 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:36,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:36,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:36,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:36,305 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:36,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:36,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:36,305 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:36,305 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:36,305 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:36,305 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:36,305 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:36,305 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:36,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:36,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:36,305 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:36,306 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:36,306 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:36,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:36,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:36,307 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:56:36,358 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=501 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/A/4dd6571afaf2452db9dfb64517d8e2dd 2023-03-20 10:56:36,410 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:36,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:36,422 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:36,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 353 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41456 deadline: 1679309856419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:36,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:36,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:36,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:36,439 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:36,439 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:36,439 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:36,439 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:36,440 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:36,440 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:36,441 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:36,442 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:36,442 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:36,446 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:36,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:36,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:36,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:36,447 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:36,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:36,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:36,448 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:36,466 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:36,466 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:36,466 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:36,466 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:36,466 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:36,478 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:36,478 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:36,478 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:36,478 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. started... 2023-03-20 10:56:36,478 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. as already flushing 2023-03-20 10:56:36,478 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:36,478 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:36,478 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:36,478 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:36,478 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:36,478 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:36,480 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:36,480 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:36,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:36,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:36,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:36,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:36,481 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:36,481 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:36,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:36,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:36,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:36,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:36,482 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:36,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57c30efa[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:36,483 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:36,483 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:36,485 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:36,485 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:36,486 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:36,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:36,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:36,486 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:36,486 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:36,486 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:36,487 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:36,487 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:36,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:36,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:36,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:36,488 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:36,488 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:36,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:36,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:36,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:36,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:36,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:36,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:36,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:36,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:36,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:36,491 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:36,491 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:36,491 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:36,491 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:36,491 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:36,502 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:36,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:36,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:36,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:36,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:36,503 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:36,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:36,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:36,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:36,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:36,505 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:36,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:36,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:36,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:36,506 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:36,506 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:36,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:36,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:36,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:36,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:36,510 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:36,510 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:36,510 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:36,510 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:36,510 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:36,510 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:36,510 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:36,510 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:36,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:36,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:36,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:36,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:36,511 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:36,514 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:36,514 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:36,530 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:36,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 355 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41456 deadline: 1679309856529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:36,614 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:36,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:36,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:36,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:36,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:36,636 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:36,636 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:36,636 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:36,636 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:36,637 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:36,637 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:36,641 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:36,641 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:36,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:36,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:36,642 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:36,642 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:36,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:36,642 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:36,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:36,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:36,644 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:36,648 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:36,648 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:36,648 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:36,648 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:36,648 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:36,658 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:36,658 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:36,667 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:36,667 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. started... 2023-03-20 10:56:36,667 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. as already flushing 2023-03-20 10:56:36,667 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:36,667 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:36,668 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:36,668 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:36,668 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:36,668 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:36,669 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:36,669 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:36,669 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:36,670 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:36,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:36,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:36,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:36,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:36,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:36,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:36,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:36,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:36,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:36,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c54db06[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:36,674 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:36,674 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:36,685 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:36,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:36,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:36,686 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:36,686 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:36,686 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:36,689 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:36,689 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:36,689 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:36,689 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:36,689 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:36,690 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:36,690 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:36,690 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:36,690 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:36,690 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:36,690 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:36,692 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:36,692 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:36,692 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:36,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:36,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:36,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:36,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:36,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:36,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:36,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:36,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:36,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:36,699 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:36,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:36,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:36,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:36,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:36,714 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:36,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:36,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:36,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:36,745 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:36,745 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:36,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:36,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:36,746 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:36,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 357 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41456 deadline: 1679309856745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:36,749 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:36,750 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:36,750 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:36,750 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:36,750 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:36,750 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:36,750 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:36,750 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:36,750 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:36,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:36,751 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:36,751 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:36,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:36,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:36,841 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=501 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/B/50eee295b73440a1ab45e8c57ce15d31 2023-03-20 10:56:36,851 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:36,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:36,855 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:36,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:36,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:36,870 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:36,870 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:36,870 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:36,871 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:36,874 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:36,874 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:36,877 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:36,877 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:36,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:36,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:36,886 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:36,886 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:36,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:36,887 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:36,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:36,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:36,888 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:36,906 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:36,906 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:36,906 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:36,906 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:36,906 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:36,918 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:36,918 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:36,930 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:36,930 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. started... 2023-03-20 10:56:36,930 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. as already flushing 2023-03-20 10:56:36,930 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:36,930 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:36,931 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:36,931 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:36,931 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:36,931 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:36,933 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:36,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:36,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:36,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:36,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:36,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:36,934 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:36,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:36,934 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:36,934 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:36,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:36,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:36,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:36,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a4aab34[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:36,939 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:36,939 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:36,945 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:36,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:36,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:36,945 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:36,945 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:36,945 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:36,946 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:36,946 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:36,946 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:36,950 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:36,950 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:36,950 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:36,950 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:36,950 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:36,950 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:36,950 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:36,950 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:36,952 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:36,952 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:36,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:36,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:36,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:36,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:36,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:36,953 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:36,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:36,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:36,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:36,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:36,953 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:36,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:36,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:36,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:36,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:36,956 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:36,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:36,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:36,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:36,968 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:36,968 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:36,968 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:36,968 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:36,968 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:36,969 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:36,969 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:36,969 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:36,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:36,968 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:36,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:36,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:36,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:36,970 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:36,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:36,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:36,970 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:36,971 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:37,095 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:37,096 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:37,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 359 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41456 deadline: 1679309857096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:37,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:37,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:37,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:37,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:37,110 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:37,110 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:37,110 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:37,110 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:37,110 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:37,110 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:37,118 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:37,118 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:37,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:37,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:37,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:37,120 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:37,120 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:37,122 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:37,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:37,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:37,123 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:37,124 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:37,124 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:37,124 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:37,124 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:37,124 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:37,124 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:37,124 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:37,124 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. started... 2023-03-20 10:56:37,125 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. as already flushing 2023-03-20 10:56:37,125 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:37,125 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:37,125 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:37,125 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:37,125 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:37,125 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:37,127 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:37,127 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:37,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:37,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:37,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:37,128 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:37,128 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:37,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:37,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:37,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:37,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:37,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:37,144 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:37,144 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@357f349b[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:37,144 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:37,144 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:37,145 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:37,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:37,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:37,145 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:37,145 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:37,145 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:37,145 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:37,145 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:37,146 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:37,146 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:37,146 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:37,146 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:37,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:37,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:37,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:37,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:37,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:37,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:37,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:37,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:37,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:37,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:37,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:37,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:37,149 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:37,149 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:37,149 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:37,149 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:37,149 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:37,150 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:37,150 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:37,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:37,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:37,150 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:37,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:37,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:37,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:37,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:37,151 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:37,152 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:37,154 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:37,159 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:37,159 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:37,159 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:37,159 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:37,159 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:37,159 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:37,159 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:37,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:37,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:37,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:37,154 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:37,162 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:37,162 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:37,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:37,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:37,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:37,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:37,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:37,168 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:56:37,262 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:37,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:37,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:37,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:37,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:37,271 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:37,271 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:37,271 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:37,271 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:37,272 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:37,272 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:37,273 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:37,273 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:37,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:37,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:37,276 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:37,276 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:37,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:37,277 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:37,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:37,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:37,277 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:37,278 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:37,278 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:37,278 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:37,278 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:37,278 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:37,279 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:37,279 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:37,279 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:37,279 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. started... 2023-03-20 10:56:37,280 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. as already flushing 2023-03-20 10:56:37,280 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:37,280 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:37,280 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:37,280 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:37,280 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:37,280 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:37,286 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:37,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:37,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:37,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:37,287 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:37,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:37,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:37,288 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:37,288 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:37,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:37,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:37,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:37,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:37,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a844810[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:37,289 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:37,289 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:37,294 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:37,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:37,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:37,294 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:37,294 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:37,294 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:37,295 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:37,296 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:37,296 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:37,296 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:37,296 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:37,296 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:37,296 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:37,296 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:37,296 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:37,296 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:37,296 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:37,298 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:37,298 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:37,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:37,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:37,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:37,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:37,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:37,299 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:37,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:37,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:37,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:37,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:37,300 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:37,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:37,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:37,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:37,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:37,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:37,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:37,302 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:37,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:37,311 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:37,311 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:37,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:37,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:37,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:37,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:37,312 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:37,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:37,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:37,312 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:37,312 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:37,312 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:37,312 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:37,312 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:37,312 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:37,312 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:37,313 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:37,313 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:37,355 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=501 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/C/4e75bb1db9f9459580b2034fb15ad88b 2023-03-20 10:56:37,374 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/A/4dd6571afaf2452db9dfb64517d8e2dd as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/4dd6571afaf2452db9dfb64517d8e2dd 2023-03-20 10:56:37,395 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/4dd6571afaf2452db9dfb64517d8e2dd, entries=200, sequenceid=501, filesize=14.3 K 2023-03-20 10:56:37,396 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/B/50eee295b73440a1ab45e8c57ce15d31 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/50eee295b73440a1ab45e8c57ce15d31 2023-03-20 10:56:37,402 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/50eee295b73440a1ab45e8c57ce15d31, entries=150, sequenceid=501, filesize=11.9 K 2023-03-20 10:56:37,403 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/C/4e75bb1db9f9459580b2034fb15ad88b as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/4e75bb1db9f9459580b2034fb15ad88b 2023-03-20 10:56:37,413 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:37,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:37,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:37,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:37,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:37,430 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:37,430 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:37,430 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:37,430 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:37,431 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:37,431 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:37,430 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/4e75bb1db9f9459580b2034fb15ad88b, entries=150, sequenceid=501, filesize=11.9 K 2023-03-20 10:56:37,432 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:37,432 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:37,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:37,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:37,432 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for c79c35ba58052cacab6709e8682a60f1 in 1141ms, sequenceid=501, compaction requested=true 2023-03-20 10:56:37,432 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c79c35ba58052cacab6709e8682a60f1: 2023-03-20 10:56:37,432 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:56:37,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:37,433 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 10:56:37,433 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:56:37,432 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 10:56:37,433 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:37,433 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:37,433 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:37,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:37,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:37,434 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:37,441 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:37,441 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:37,441 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:37,441 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:37,441 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:37,445 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39782 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 10:56:37,445 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1912): c79c35ba58052cacab6709e8682a60f1/A is initiating minor compaction (all files) 2023-03-20 10:56:37,445 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c79c35ba58052cacab6709e8682a60f1/A in TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:37,445 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/942c1a5846f14fcc9bc9f5cd62570a29, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/cd3b7cc9b2a24779810b424c3bf72a37, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/4dd6571afaf2452db9dfb64517d8e2dd] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp, totalSize=38.8 K 2023-03-20 10:56:37,445 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37342 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 10:56:37,446 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): c79c35ba58052cacab6709e8682a60f1/B is initiating minor compaction (all files) 2023-03-20 10:56:37,446 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c79c35ba58052cacab6709e8682a60f1/B in TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:37,446 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/2b0dde7e4974450c99f04e9867a2ffd3, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/5838a70c16524ac49b30fd9a085bde64, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/50eee295b73440a1ab45e8c57ce15d31] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp, totalSize=36.5 K 2023-03-20 10:56:37,446 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 942c1a5846f14fcc9bc9f5cd62570a29, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=465, earliestPutTs=1679309793134 2023-03-20 10:56:37,447 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:37,447 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:37,447 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:37,447 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. started... 2023-03-20 10:56:37,448 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c79c35ba58052cacab6709e8682a60f1 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-20 10:56:37,448 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 2b0dde7e4974450c99f04e9867a2ffd3, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=465, earliestPutTs=1679309793134 2023-03-20 10:56:37,448 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=A 2023-03-20 10:56:37,448 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting cd3b7cc9b2a24779810b424c3bf72a37, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=482, earliestPutTs=1679309793924 2023-03-20 10:56:37,448 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 5838a70c16524ac49b30fd9a085bde64, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=482, earliestPutTs=1679309793924 2023-03-20 10:56:37,448 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 4dd6571afaf2452db9dfb64517d8e2dd, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=501, earliestPutTs=1679309795115 2023-03-20 10:56:37,449 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 50eee295b73440a1ab45e8c57ce15d31, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=501, earliestPutTs=1679309795115 2023-03-20 10:56:37,433 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:56:37,448 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:37,454 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=B 2023-03-20 10:56:37,455 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:37,455 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=C 2023-03-20 10:56:37,455 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:37,463 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] throttle.PressureAwareThroughputController(145): c79c35ba58052cacab6709e8682a60f1#A#compaction#78 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:56:37,463 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): c79c35ba58052cacab6709e8682a60f1#B#compaction#79 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:56:37,534 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/A/6a279bc773bf4eb0903089ce5bb7da77 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/6a279bc773bf4eb0903089ce5bb7da77 2023-03-20 10:56:37,546 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=515 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/A/5f9bc8713b694de8895d88b3e1aafbaf 2023-03-20 10:56:37,547 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/B/ab28eaf508db4f5ca916dc619f2b74c5 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/ab28eaf508db4f5ca916dc619f2b74c5 2023-03-20 10:56:37,563 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c79c35ba58052cacab6709e8682a60f1/A of c79c35ba58052cacab6709e8682a60f1 into 6a279bc773bf4eb0903089ce5bb7da77(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:56:37,563 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c79c35ba58052cacab6709e8682a60f1: 2023-03-20 10:56:37,563 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1., storeName=c79c35ba58052cacab6709e8682a60f1/A, priority=13, startTime=1679309797432; duration=0sec 2023-03-20 10:56:37,564 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:56:37,564 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 10:56:37,566 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37342 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 10:56:37,566 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1912): c79c35ba58052cacab6709e8682a60f1/C is initiating minor compaction (all files) 2023-03-20 10:56:37,566 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c79c35ba58052cacab6709e8682a60f1/C in TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:37,566 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/a1181ab928014935aafea6b32f9cf36b, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/b1ade1a10b5e4669bf491d676c5cffb4, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/4e75bb1db9f9459580b2034fb15ad88b] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp, totalSize=36.5 K 2023-03-20 10:56:37,567 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting a1181ab928014935aafea6b32f9cf36b, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=465, earliestPutTs=1679309793134 2023-03-20 10:56:37,568 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting b1ade1a10b5e4669bf491d676c5cffb4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=482, earliestPutTs=1679309793924 2023-03-20 10:56:37,569 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 4e75bb1db9f9459580b2034fb15ad88b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=501, earliestPutTs=1679309795115 2023-03-20 10:56:37,570 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c79c35ba58052cacab6709e8682a60f1/B of c79c35ba58052cacab6709e8682a60f1 into ab28eaf508db4f5ca916dc619f2b74c5(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:56:37,570 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c79c35ba58052cacab6709e8682a60f1: 2023-03-20 10:56:37,570 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1., storeName=c79c35ba58052cacab6709e8682a60f1/B, priority=13, startTime=1679309797432; duration=0sec 2023-03-20 10:56:37,570 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:56:37,591 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] throttle.PressureAwareThroughputController(145): c79c35ba58052cacab6709e8682a60f1#C#compaction#82 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:56:37,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on c79c35ba58052cacab6709e8682a60f1 2023-03-20 10:56:37,616 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. as already flushing 2023-03-20 10:56:37,693 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:37,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 376 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41456 deadline: 1679309857692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:37,809 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:37,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 378 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41456 deadline: 1679309857803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:38,011 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=515 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/B/640150bfc6f64975b37c7ee0f725624a 2023-03-20 10:56:38,031 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/C/332b2db836f44ec19a3c283936198e1d as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/332b2db836f44ec19a3c283936198e1d 2023-03-20 10:56:38,032 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:38,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 380 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41456 deadline: 1679309858014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:38,058 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c79c35ba58052cacab6709e8682a60f1/C of c79c35ba58052cacab6709e8682a60f1 into 332b2db836f44ec19a3c283936198e1d(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:56:38,058 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c79c35ba58052cacab6709e8682a60f1: 2023-03-20 10:56:38,059 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1., storeName=c79c35ba58052cacab6709e8682a60f1/C, priority=13, startTime=1679309797433; duration=0sec 2023-03-20 10:56:38,059 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:56:38,355 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:38,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 382 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41456 deadline: 1679309858351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:38,484 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=515 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/C/7463bc21680c4c898d85662db3a8d0b5 2023-03-20 10:56:38,491 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/A/5f9bc8713b694de8895d88b3e1aafbaf as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/5f9bc8713b694de8895d88b3e1aafbaf 2023-03-20 10:56:38,497 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/5f9bc8713b694de8895d88b3e1aafbaf, entries=150, sequenceid=515, filesize=11.9 K 2023-03-20 10:56:38,498 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/B/640150bfc6f64975b37c7ee0f725624a as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/640150bfc6f64975b37c7ee0f725624a 2023-03-20 10:56:38,508 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/640150bfc6f64975b37c7ee0f725624a, entries=150, sequenceid=515, filesize=11.9 K 2023-03-20 10:56:38,509 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/C/7463bc21680c4c898d85662db3a8d0b5 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/7463bc21680c4c898d85662db3a8d0b5 2023-03-20 10:56:38,515 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/7463bc21680c4c898d85662db3a8d0b5, entries=150, sequenceid=515, filesize=11.9 K 2023-03-20 10:56:38,516 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for c79c35ba58052cacab6709e8682a60f1 in 1069ms, sequenceid=515, compaction requested=false 2023-03-20 10:56:38,516 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c79c35ba58052cacab6709e8682a60f1: 2023-03-20 10:56:38,516 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:38,516 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:38,516 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:38,516 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:38,517 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:38,517 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:38,518 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:38,519 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:38,519 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:38,519 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:38,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:38,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:38,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:38,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:38,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:38,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:38,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:38,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:38,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:38,523 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:38,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@502230ad[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:38,523 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:38,524 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:38,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:38,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:38,524 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:38,524 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:38,524 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:38,524 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:38,524 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:38,527 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:38,527 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:38,527 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:38,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:38,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:38,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:38,527 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:38,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:38,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:38,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:38,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:38,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:38,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:38,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:38,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:38,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:38,530 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:38,530 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:38,530 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:38,530 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:38,530 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:38,543 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:38,543 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:38,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:38,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:38,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:38,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:38,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:38,547 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:38,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:38,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:38,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:38,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:38,547 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:38,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:38,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:38,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:38,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:38,548 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:38,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:38,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:38,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:38,552 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:38,552 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:38,552 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:38,552 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:38,552 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:38,552 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:38,552 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:38,552 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:38,553 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:38,553 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:38,553 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:38,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:38,553 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:38,553 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:38,552 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:38,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:38,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:38,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:38,653 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:38,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:38,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:38,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:38,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:38,668 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:38,668 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:38,668 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:38,668 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:38,670 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:38,670 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:38,672 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:38,673 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:38,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:38,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:38,673 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:38,674 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:38,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:38,674 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:38,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:38,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:38,674 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:38,678 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:38,678 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:38,678 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:38,678 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:38,678 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:38,680 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:38,680 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:38,680 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:38,689 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. started... 2023-03-20 10:56:38,690 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c79c35ba58052cacab6709e8682a60f1 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-20 10:56:38,691 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=A 2023-03-20 10:56:38,691 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:38,691 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=B 2023-03-20 10:56:38,691 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:38,692 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=C 2023-03-20 10:56:38,692 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:38,701 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=536 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/A/f138c61dd4bd4c5dbf2d83e1de80bce3 2023-03-20 10:56:38,748 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=536 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/B/313c935cf3774993baeef8013222033b 2023-03-20 10:56:38,792 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=536 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/C/829d43168bf349c1adece151d011b160 2023-03-20 10:56:38,807 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/A/f138c61dd4bd4c5dbf2d83e1de80bce3 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/f138c61dd4bd4c5dbf2d83e1de80bce3 2023-03-20 10:56:38,816 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/f138c61dd4bd4c5dbf2d83e1de80bce3, entries=150, sequenceid=536, filesize=11.9 K 2023-03-20 10:56:38,817 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/B/313c935cf3774993baeef8013222033b as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/313c935cf3774993baeef8013222033b 2023-03-20 10:56:38,828 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/313c935cf3774993baeef8013222033b, entries=150, sequenceid=536, filesize=11.9 K 2023-03-20 10:56:38,830 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/C/829d43168bf349c1adece151d011b160 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/829d43168bf349c1adece151d011b160 2023-03-20 10:56:38,836 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/829d43168bf349c1adece151d011b160, entries=150, sequenceid=536, filesize=11.9 K 2023-03-20 10:56:38,837 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=0 B/0 for c79c35ba58052cacab6709e8682a60f1 in 147ms, sequenceid=536, compaction requested=true 2023-03-20 10:56:38,837 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c79c35ba58052cacab6709e8682a60f1: 2023-03-20 10:56:38,838 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:38,838 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:38,838 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:38,838 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:38,838 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:38,838 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:38,843 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:38,846 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:38,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:38,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:38,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:38,846 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:38,846 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:38,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:38,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:38,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:38,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:38,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:38,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:38,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2fce3296[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:38,848 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:38,850 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:38,860 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:38,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:38,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:38,860 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:38,860 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:38,860 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:38,860 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:38,861 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:38,869 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:38,869 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:38,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:38,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:38,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:38,869 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:38,869 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:38,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:38,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:38,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:38,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:38,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:38,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:38,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:38,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on c79c35ba58052cacab6709e8682a60f1 2023-03-20 10:56:38,873 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c79c35ba58052cacab6709e8682a60f1 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-20 10:56:38,873 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=A 2023-03-20 10:56:38,873 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:38,873 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=B 2023-03-20 10:56:38,873 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:38,873 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=C 2023-03-20 10:56:38,874 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:38,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:38,874 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:38,874 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:38,874 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:38,874 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:38,874 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:38,874 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:38,877 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:38,877 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:38,877 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:38,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:38,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:38,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:38,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:38,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:38,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:38,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:38,878 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:38,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:38,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:38,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:38,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:38,880 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:38,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:38,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:38,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:38,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:38,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:38,888 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:38,888 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:38,888 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:38,889 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:38,889 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:38,889 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:38,889 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:38,889 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:38,889 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:38,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:38,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:38,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:38,889 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:38,889 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:38,890 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:56:38,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:38,889 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:38,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:38,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:38,904 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=546 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/A/747eb8fa964a414cbeeb03b3afdf45c9 2023-03-20 10:56:38,934 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=546 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/B/cf75be2f3a904225b3477df831fd3381 2023-03-20 10:56:38,978 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679309798946, max timestamp=1679309798976Num uniques 859; 2023-03-20 10:56:38,978 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-20 10:56:38,979 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679309798946, max timestamp=1679309798976Num uniques 859; 2023-03-20 10:56:38,979 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-20 10:56:38,980 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679309798946, max timestamp=1679309798976Num uniques 859; 2023-03-20 10:56:38,980 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-20 10:56:38,990 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:38,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:38,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:38,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:38,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:38,997 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:38,997 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:38,997 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:38,997 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:38,997 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:38,998 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:38,999 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:38,999 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:38,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:38,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:38,999 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:38,999 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:38,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:39,000 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:39,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:39,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:39,000 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:39,023 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:39,023 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:39,023 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:39,023 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:39,023 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:39,023 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:39,023 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:39,023 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:39,023 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. started... 2023-03-20 10:56:39,023 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. as already flushing 2023-03-20 10:56:39,023 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:39,024 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:39,024 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:39,024 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,024 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:39,024 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:39,024 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=546 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/C/514fbc06db584f37bd70e3ef4a6292fc 2023-03-20 10:56:39,025 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,025 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:39,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:39,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:39,026 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:39,026 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:39,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:39,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:39,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:39,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:39,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:39,029 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:39,029 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:39,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63590018[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:39,032 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,032 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:39,035 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/A/747eb8fa964a414cbeeb03b3afdf45c9 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/747eb8fa964a414cbeeb03b3afdf45c9 2023-03-20 10:56:39,039 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:39,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:39,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:39,039 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:39,039 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:39,039 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:39,044 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,044 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:39,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:39,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:39,044 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:39,044 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:39,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:39,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:39,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:39,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:39,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:39,048 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/747eb8fa964a414cbeeb03b3afdf45c9, entries=100, sequenceid=546, filesize=9.5 K 2023-03-20 10:56:39,049 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/B/cf75be2f3a904225b3477df831fd3381 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/cf75be2f3a904225b3477df831fd3381 2023-03-20 10:56:39,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:39,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:39,051 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:39,051 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:39,051 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:39,051 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:39,051 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:39,054 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/cf75be2f3a904225b3477df831fd3381, entries=100, sequenceid=546, filesize=9.5 K 2023-03-20 10:56:39,056 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/C/514fbc06db584f37bd70e3ef4a6292fc as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/514fbc06db584f37bd70e3ef4a6292fc 2023-03-20 10:56:39,062 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:39,062 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:39,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:39,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:39,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:39,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:39,063 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:39,064 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,066 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:39,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:39,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:39,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:39,070 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:39,070 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:39,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:39,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:39,070 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,070 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:39,070 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:39,070 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,070 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:39,070 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:39,070 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:39,070 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:39,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:39,070 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:39,071 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:39,071 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:39,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:39,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:39,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:39,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:39,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:39,072 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:56:39,078 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/514fbc06db584f37bd70e3ef4a6292fc, entries=100, sequenceid=546, filesize=9.5 K 2023-03-20 10:56:39,079 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=127.47 KB/130530 for c79c35ba58052cacab6709e8682a60f1 in 207ms, sequenceid=546, compaction requested=true 2023-03-20 10:56:39,079 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c79c35ba58052cacab6709e8682a60f1: 2023-03-20 10:56:39,079 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 10:56:39,079 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:56:39,079 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 10:56:39,080 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:56:39,080 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:56:39,081 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47186 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 10:56:39,081 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): c79c35ba58052cacab6709e8682a60f1/B is initiating minor compaction (all files) 2023-03-20 10:56:39,082 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c79c35ba58052cacab6709e8682a60f1/B in TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:39,082 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/ab28eaf508db4f5ca916dc619f2b74c5, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/640150bfc6f64975b37c7ee0f725624a, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/313c935cf3774993baeef8013222033b, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/cf75be2f3a904225b3477df831fd3381] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp, totalSize=46.1 K 2023-03-20 10:56:39,082 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47186 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 10:56:39,082 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1912): c79c35ba58052cacab6709e8682a60f1/A is initiating minor compaction (all files) 2023-03-20 10:56:39,082 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c79c35ba58052cacab6709e8682a60f1/A in TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:39,082 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/6a279bc773bf4eb0903089ce5bb7da77, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/5f9bc8713b694de8895d88b3e1aafbaf, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/f138c61dd4bd4c5dbf2d83e1de80bce3, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/747eb8fa964a414cbeeb03b3afdf45c9] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp, totalSize=46.1 K 2023-03-20 10:56:39,083 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting ab28eaf508db4f5ca916dc619f2b74c5, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=501, earliestPutTs=1679309795115 2023-03-20 10:56:39,084 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 6a279bc773bf4eb0903089ce5bb7da77, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=501, earliestPutTs=1679309795115 2023-03-20 10:56:39,084 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 640150bfc6f64975b37c7ee0f725624a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=515, earliestPutTs=1679309796371 2023-03-20 10:56:39,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on c79c35ba58052cacab6709e8682a60f1 2023-03-20 10:56:39,086 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c79c35ba58052cacab6709e8682a60f1 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-03-20 10:56:39,086 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=A 2023-03-20 10:56:39,086 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:56:39,086 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=B 2023-03-20 10:56:39,086 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:56:39,086 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=C 2023-03-20 10:56:39,086 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:56:39,088 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 313c935cf3774993baeef8013222033b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=536, earliestPutTs=1679309797631 2023-03-20 10:56:39,088 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 5f9bc8713b694de8895d88b3e1aafbaf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=515, earliestPutTs=1679309796371 2023-03-20 10:56:39,089 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting cf75be2f3a904225b3477df831fd3381, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=546, earliestPutTs=1679309798865 2023-03-20 10:56:39,090 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting f138c61dd4bd4c5dbf2d83e1de80bce3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=536, earliestPutTs=1679309797631 2023-03-20 10:56:39,090 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 747eb8fa964a414cbeeb03b3afdf45c9, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=546, earliestPutTs=1679309798865 2023-03-20 10:56:39,111 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): c79c35ba58052cacab6709e8682a60f1#B#compaction#91 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:56:39,115 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] throttle.PressureAwareThroughputController(145): c79c35ba58052cacab6709e8682a60f1#A#compaction#92 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:56:39,144 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:39,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 431 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41456 deadline: 1679309859140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,171 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:39,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:39,179 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:39,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:39,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:39,200 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:39,200 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:39,200 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:39,200 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:39,201 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:39,201 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:39,202 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=582 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/A/2d8b8661bde740148d10231de04c2bf7 2023-03-20 10:56:39,202 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:39,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:39,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:39,202 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:39,202 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,203 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:39,203 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:39,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:39,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:39,203 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:39,214 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:39,214 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:39,214 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:39,214 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:39,214 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:39,216 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:39,216 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:39,216 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:39,216 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. started... 2023-03-20 10:56:39,216 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. as already flushing 2023-03-20 10:56:39,216 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:39,216 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:39,216 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:39,216 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,216 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:39,216 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:39,227 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,227 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:39,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:39,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:39,227 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:39,227 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:39,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:39,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:39,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:39,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:39,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:39,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b4500ab[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:39,237 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:39,237 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:39,239 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:39,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:39,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:39,239 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:39,239 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:39,239 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:39,240 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,240 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:39,241 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,241 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:39,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,241 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:39,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:39,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:39,241 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:39,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:39,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:39,243 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/B/375926c82d3545b9b229022aaa924921 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/375926c82d3545b9b229022aaa924921 2023-03-20 10:56:39,243 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/A/47a0379ab76f4631a18c34a2d156e704 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/47a0379ab76f4631a18c34a2d156e704 2023-03-20 10:56:39,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:39,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:39,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:39,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:39,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:39,246 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:39,246 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:39,246 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:39,246 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:39,247 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:39,248 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:39,248 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:39,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:39,248 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:39,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:39,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:39,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:39,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:39,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:39,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:39,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:39,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:39,250 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:39,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 433 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41456 deadline: 1679309859246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:39,252 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:39,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:39,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,254 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:39,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:39,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,256 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c79c35ba58052cacab6709e8682a60f1/A of c79c35ba58052cacab6709e8682a60f1 into 47a0379ab76f4631a18c34a2d156e704(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:56:39,256 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c79c35ba58052cacab6709e8682a60f1: 2023-03-20 10:56:39,256 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1., storeName=c79c35ba58052cacab6709e8682a60f1/A, priority=12, startTime=1679309799079; duration=0sec 2023-03-20 10:56:39,256 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:56:39,256 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 10:56:39,256 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c79c35ba58052cacab6709e8682a60f1/B of c79c35ba58052cacab6709e8682a60f1 into 375926c82d3545b9b229022aaa924921(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:56:39,256 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c79c35ba58052cacab6709e8682a60f1: 2023-03-20 10:56:39,256 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1., storeName=c79c35ba58052cacab6709e8682a60f1/B, priority=12, startTime=1679309799079; duration=0sec 2023-03-20 10:56:39,257 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:56:39,260 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47186 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 10:56:39,260 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1912): c79c35ba58052cacab6709e8682a60f1/C is initiating minor compaction (all files) 2023-03-20 10:56:39,260 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c79c35ba58052cacab6709e8682a60f1/C in TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:39,260 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/332b2db836f44ec19a3c283936198e1d, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/7463bc21680c4c898d85662db3a8d0b5, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/829d43168bf349c1adece151d011b160, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/514fbc06db584f37bd70e3ef4a6292fc] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp, totalSize=46.1 K 2023-03-20 10:56:39,261 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 332b2db836f44ec19a3c283936198e1d, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=501, earliestPutTs=1679309795115 2023-03-20 10:56:39,261 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 7463bc21680c4c898d85662db3a8d0b5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=515, earliestPutTs=1679309796371 2023-03-20 10:56:39,262 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:39,262 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:39,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:39,262 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:56:39,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:39,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:39,262 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:39,262 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:39,262 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:39,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:39,262 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 829d43168bf349c1adece151d011b160, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=536, earliestPutTs=1679309797631 2023-03-20 10:56:39,262 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:39,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:39,262 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,262 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:39,262 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:39,262 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:39,262 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:39,262 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:39,263 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 514fbc06db584f37bd70e3ef4a6292fc, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=546, earliestPutTs=1679309798865 2023-03-20 10:56:39,282 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] throttle.PressureAwareThroughputController(145): c79c35ba58052cacab6709e8682a60f1#C#compaction#94 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:56:39,339 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/C/4277964fb0004c95bd2c5e12a20b92f8 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/4277964fb0004c95bd2c5e12a20b92f8 2023-03-20 10:56:39,347 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c79c35ba58052cacab6709e8682a60f1/C of c79c35ba58052cacab6709e8682a60f1 into 4277964fb0004c95bd2c5e12a20b92f8(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:56:39,347 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c79c35ba58052cacab6709e8682a60f1: 2023-03-20 10:56:39,347 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1., storeName=c79c35ba58052cacab6709e8682a60f1/C, priority=12, startTime=1679309799080; duration=0sec 2023-03-20 10:56:39,347 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:56:39,362 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:39,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:39,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:39,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:39,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:39,372 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:39,372 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:39,372 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:39,372 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:39,373 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:39,373 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:39,374 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:39,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:39,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:39,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:39,375 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:39,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:39,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:39,375 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:39,375 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,376 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,376 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:39,376 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:39,376 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:39,376 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:39,376 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:39,377 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:39,377 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:39,377 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:39,377 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:39,377 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. started... 2023-03-20 10:56:39,377 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. as already flushing 2023-03-20 10:56:39,377 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:39,377 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:39,377 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:39,377 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,377 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:39,377 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:39,379 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:39,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:39,379 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:39,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:39,380 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:39,380 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:39,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:39,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:39,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:39,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:39,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a23e836[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:39,381 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:39,381 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:39,382 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:39,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:39,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:39,382 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:39,383 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:39,383 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:39,384 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:39,384 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:39,384 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:39,384 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,386 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:39,386 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:39,386 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:39,386 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:39,386 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:39,386 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:39,386 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:39,388 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:39,388 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:39,388 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:39,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:39,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:39,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:39,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:39,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:39,389 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:39,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:39,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:39,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:39,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:39,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:39,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:39,392 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:39,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:39,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,415 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,415 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:39,415 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:39,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:39,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:39,415 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,416 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:39,416 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:39,416 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:39,415 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:39,416 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:39,416 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:39,418 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:39,418 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:39,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:39,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:39,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:39,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:39,455 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:39,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 435 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41456 deadline: 1679309859453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,516 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:39,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:39,519 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:39,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:39,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:39,534 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:39,534 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:39,534 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:39,534 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:39,538 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:39,538 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:39,539 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:39,539 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:39,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:39,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:39,540 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:39,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:39,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:39,540 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:39,542 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,542 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:39,542 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:39,542 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:39,542 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:39,542 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:39,542 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:39,554 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:39,554 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:39,554 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:39,554 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. started... 2023-03-20 10:56:39,554 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. as already flushing 2023-03-20 10:56:39,554 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:39,554 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:39,554 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:39,554 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,554 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:39,555 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:39,569 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:39,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:39,569 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:39,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:39,570 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:39,571 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:39,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:39,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:39,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:39,574 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:39,574 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24273f66[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:39,574 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:39,574 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:39,577 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,577 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:39,578 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:39,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:39,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:39,578 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:39,578 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:39,578 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:39,579 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:39,579 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:39,579 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:39,581 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:39,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:39,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:39,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:39,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:39,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:39,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:39,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:39,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:39,584 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:39,584 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:39,584 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:39,584 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:39,584 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:39,602 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:39,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:39,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:39,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:39,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:39,603 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:39,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:39,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:39,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,604 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:39,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:39,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,606 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:39,606 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:39,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:39,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:39,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:39,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:39,607 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,607 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:39,607 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:39,608 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:39,608 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:39,608 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:39,608 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:39,610 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:39,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:39,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:39,607 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:39,613 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,613 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:39,613 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:39,613 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:39,708 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:39,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:39,710 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=582 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/B/f7fc420850f14bb5b4fcfc79f6c9c0bf 2023-03-20 10:56:39,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:39,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:39,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:39,722 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:39,722 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:39,722 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:39,722 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:39,723 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:39,723 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:39,723 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,724 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,724 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:39,726 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:39,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:39,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:39,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:39,727 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:39,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:39,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:39,728 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:39,730 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:39,730 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:39,730 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:39,730 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:39,730 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:39,731 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:39,731 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:39,731 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:39,731 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. started... 2023-03-20 10:56:39,732 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. as already flushing 2023-03-20 10:56:39,732 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:39,732 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:39,732 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:39,732 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,742 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:39,742 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:39,743 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:39,743 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:39,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:39,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:39,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:39,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:39,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:39,745 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:39,746 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:39,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:39,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@fd5a23a[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:39,746 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:39,746 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:39,747 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,747 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:39,750 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:39,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:39,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:39,750 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:39,750 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:39,750 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:39,759 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=582 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/C/56488fdb03274d96a17759a6cbfeffe9 2023-03-20 10:56:39,760 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:39,760 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:39,760 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:39,761 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:39,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:39,763 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:39,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:39,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 437 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41456 deadline: 1679309859759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:39,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:39,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:39,768 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/A/2d8b8661bde740148d10231de04c2bf7 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/2d8b8661bde740148d10231de04c2bf7 2023-03-20 10:56:39,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:39,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:39,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:39,790 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:39,790 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:39,790 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:39,790 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:39,790 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:39,797 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/2d8b8661bde740148d10231de04c2bf7, entries=200, sequenceid=582, filesize=14.3 K 2023-03-20 10:56:39,798 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/B/f7fc420850f14bb5b4fcfc79f6c9c0bf as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/f7fc420850f14bb5b4fcfc79f6c9c0bf 2023-03-20 10:56:39,800 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:39,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:39,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:39,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:39,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:39,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:39,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:39,801 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:39,801 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:39,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:39,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:39,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:39,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:39,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:39,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:39,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:39,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,806 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,806 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/f7fc420850f14bb5b4fcfc79f6c9c0bf, entries=150, sequenceid=582, filesize=11.9 K 2023-03-20 10:56:39,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:39,810 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,814 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/C/56488fdb03274d96a17759a6cbfeffe9 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/56488fdb03274d96a17759a6cbfeffe9 2023-03-20 10:56:39,816 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,817 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:39,817 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:39,817 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,817 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:39,817 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:39,817 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:39,817 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:39,817 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:39,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:39,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:39,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:39,818 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:39,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:39,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:39,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:39,826 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:39,826 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:39,838 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/56488fdb03274d96a17759a6cbfeffe9, entries=150, sequenceid=582, filesize=11.9 K 2023-03-20 10:56:39,840 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for c79c35ba58052cacab6709e8682a60f1 in 754ms, sequenceid=582, compaction requested=false 2023-03-20 10:56:39,840 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c79c35ba58052cacab6709e8682a60f1: 2023-03-20 10:56:39,930 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:39,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:39,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:39,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:39,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:39,990 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:39,990 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:39,990 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:39,990 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:39,994 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:39,994 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:39,995 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,999 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:39,999 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:39,999 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:40,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:40,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:40,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:40,010 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:40,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:40,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:40,011 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:40,026 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:40,026 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:40,026 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:40,026 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:40,026 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:40,038 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:40,038 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:40,044 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:40,045 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. started... 2023-03-20 10:56:40,045 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c79c35ba58052cacab6709e8682a60f1 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-20 10:56:40,045 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=A 2023-03-20 10:56:40,045 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:40,045 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=B 2023-03-20 10:56:40,045 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:40,045 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=C 2023-03-20 10:56:40,046 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:40,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on c79c35ba58052cacab6709e8682a60f1 2023-03-20 10:56:40,270 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. as already flushing 2023-03-20 10:56:40,345 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679309800323, max timestamp=1679309800343Num uniques 859; 2023-03-20 10:56:40,345 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-20 10:56:40,346 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679309800323, max timestamp=1679309800343Num uniques 859; 2023-03-20 10:56:40,347 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-20 10:56:40,347 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679309800323, max timestamp=1679309800343Num uniques 859; 2023-03-20 10:56:40,347 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-20 10:56:40,494 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:40,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 472 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41456 deadline: 1679309860490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:40,578 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=595 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/A/fe2b7ae0612a46fead7240fea74c2eea 2023-03-20 10:56:40,609 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:40,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 474 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41456 deadline: 1679309860600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:40,818 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:40,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 476 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41456 deadline: 1679309860814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:41,124 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:41,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 478 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41456 deadline: 1679309861122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:41,127 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=595 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/B/0e2478befe064772a46ce50444e12782 2023-03-20 10:56:41,544 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=595 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/C/cd398a19719d45618bf95e294f0bea7b 2023-03-20 10:56:41,563 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/A/fe2b7ae0612a46fead7240fea74c2eea as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/fe2b7ae0612a46fead7240fea74c2eea 2023-03-20 10:56:41,578 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/fe2b7ae0612a46fead7240fea74c2eea, entries=100, sequenceid=595, filesize=9.5 K 2023-03-20 10:56:41,580 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/B/0e2478befe064772a46ce50444e12782 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/0e2478befe064772a46ce50444e12782 2023-03-20 10:56:41,592 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/0e2478befe064772a46ce50444e12782, entries=100, sequenceid=595, filesize=9.5 K 2023-03-20 10:56:41,595 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/C/cd398a19719d45618bf95e294f0bea7b as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/cd398a19719d45618bf95e294f0bea7b 2023-03-20 10:56:41,625 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/cd398a19719d45618bf95e294f0bea7b, entries=100, sequenceid=595, filesize=9.5 K 2023-03-20 10:56:41,638 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for c79c35ba58052cacab6709e8682a60f1 in 1593ms, sequenceid=595, compaction requested=true 2023-03-20 10:56:41,638 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c79c35ba58052cacab6709e8682a60f1: 2023-03-20 10:56:41,638 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:41,638 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:41,638 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:41,639 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:41,639 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:41,639 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:41,640 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:41,640 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:41,640 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:41,646 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:41,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:41,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:41,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:41,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:41,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:41,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:41,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:41,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:41,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on c79c35ba58052cacab6709e8682a60f1 2023-03-20 10:56:41,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:41,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c06ab33[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:41,648 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:41,648 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:41,650 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c79c35ba58052cacab6709e8682a60f1 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-03-20 10:56:41,650 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=A 2023-03-20 10:56:41,651 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-20 10:56:41,651 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=B 2023-03-20 10:56:41,651 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-20 10:56:41,651 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=C 2023-03-20 10:56:41,651 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-20 10:56:41,653 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:41,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:41,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:41,653 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:41,653 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:41,653 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:41,658 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:41,658 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:41,666 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:41,666 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:41,666 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:41,674 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:41,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:41,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:41,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:41,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:41,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:41,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:41,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:41,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:41,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:41,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:41,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:41,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:41,683 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:41,683 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:41,683 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:41,683 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:41,683 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:41,685 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:41,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:41,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:41,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:41,685 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=633 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/A/e42c1e789f7448629cc30ef73dbcf918 2023-03-20 10:56:41,686 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:41,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:41,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:41,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:41,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:41,719 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:41,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:41,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:41,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:41,740 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:41,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:41,742 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:41,743 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:41,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 487 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41456 deadline: 1679309861743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:41,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:41,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:41,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:41,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:41,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:41,748 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:41,748 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:41,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:41,749 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:41,749 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:41,750 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:41,750 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:41,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:41,750 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:41,750 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:41,750 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:41,750 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:41,750 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:41,750 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:41,750 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:41,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:41,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:41,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:41,847 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:41,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 489 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41456 deadline: 1679309861845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:41,857 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:41,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:41,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:41,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:41,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:41,895 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:41,895 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:41,895 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:41,895 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:41,902 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:41,902 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:41,914 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:41,914 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:41,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:41,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:41,914 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:41,914 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:41,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:41,915 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:41,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:41,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:41,915 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:41,922 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:41,922 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:41,922 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:41,922 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:41,931 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:41,931 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:41,931 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:41,931 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. started... 2023-03-20 10:56:41,931 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. as already flushing 2023-03-20 10:56:41,931 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:41,931 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:41,931 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:41,931 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:41,932 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:41,932 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:41,933 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:41,933 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:41,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:41,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:41,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:41,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:41,933 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:41,933 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:41,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:41,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:41,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:41,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:41,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:41,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@790192dd[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:41,935 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:41,935 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:41,936 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:41,936 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:41,938 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:41,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:41,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:41,938 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:41,938 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:41,938 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:41,939 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:41,939 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:41,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:41,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:41,939 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:41,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:41,940 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:41,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:41,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:41,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:41,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:41,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:41,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:41,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:41,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:41,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:41,998 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:41,998 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:41,998 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:41,998 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:41,998 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:42,010 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:42,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:42,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:42,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:42,014 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:42,014 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:42,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:42,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:42,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:42,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:42,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:42,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:42,022 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:42,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:42,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:42,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:42,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:42,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:42,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:42,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:42,026 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:42,028 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:42,028 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:42,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:42,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:42,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:42,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:42,029 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:42,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:42,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:42,030 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:42,030 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:42,030 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:42,030 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:42,030 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:42,030 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:42,030 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:42,030 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:42,030 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:42,053 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:42,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 491 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41456 deadline: 1679309862049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:42,130 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:42,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:42,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:42,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:42,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:42,146 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:42,146 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:42,146 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:42,146 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:42,147 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:42,147 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:42,181 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:42,181 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:42,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:42,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:42,182 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:42,182 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:42,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:42,182 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:42,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:42,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:42,183 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:42,183 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:42,183 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:42,183 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:42,183 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:42,183 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:42,190 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:42,190 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:42,190 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:42,191 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. started... 2023-03-20 10:56:42,191 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. as already flushing 2023-03-20 10:56:42,191 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:42,191 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:42,191 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:42,191 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:42,191 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:42,191 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:42,197 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:42,197 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:42,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:42,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:42,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:42,198 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:42,198 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:42,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:42,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:42,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:42,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:42,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:42,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:42,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64c04e94[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:42,200 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:42,200 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:42,201 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:42,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:42,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:42,201 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:42,201 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:42,201 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:42,201 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:42,201 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:42,202 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:42,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:42,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:42,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:42,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:42,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:42,203 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:42,203 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:42,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:42,203 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:42,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:42,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:42,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:42,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:42,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:42,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:42,205 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:42,205 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:42,205 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:42,205 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:42,206 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:42,207 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:42,207 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:42,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:42,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:42,207 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:42,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:42,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:42,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:42,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:42,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:42,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:42,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:42,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:42,208 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:42,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:42,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:42,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:42,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:42,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:42,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:42,210 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:42,212 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:42,212 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:42,212 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:42,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:42,212 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:42,212 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:42,212 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:42,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:42,212 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:42,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:42,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:42,212 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:42,212 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:42,213 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:42,213 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:42,213 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:42,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:42,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:42,227 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=633 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/B/d3f46b5064d743f8a642430ea9e5b067 2023-03-20 10:56:42,313 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:42,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:42,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:42,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:42,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:42,320 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:42,320 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:42,320 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:42,320 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:42,321 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:42,321 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:42,323 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:42,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:42,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:42,323 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:42,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:42,324 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:42,324 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:42,324 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:42,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:42,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:42,325 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:42,325 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:42,325 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:42,325 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:42,325 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:42,325 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:42,326 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:42,326 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:42,326 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:42,326 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. started... 2023-03-20 10:56:42,326 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. as already flushing 2023-03-20 10:56:42,326 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:42,327 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:42,327 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:42,327 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:42,327 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:42,327 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:42,329 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:42,330 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:42,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:42,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:42,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:42,330 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:42,330 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:42,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:42,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:42,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:42,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:42,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:42,334 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:42,334 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4626f444[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:42,334 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:42,334 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:42,336 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:42,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:42,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:42,336 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:42,336 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:42,336 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:42,336 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:42,336 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:42,337 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:42,337 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:42,337 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:42,337 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:42,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:42,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:42,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:42,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:42,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:42,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:42,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:42,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:42,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:42,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:42,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:42,340 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:42,340 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:42,340 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:42,340 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:42,340 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:42,340 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:42,341 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:42,341 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:42,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:42,341 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:42,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:42,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:42,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:42,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:42,342 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:42,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:42,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:42,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:42,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:42,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:42,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:42,343 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:42,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:42,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:42,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:42,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:42,345 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:42,345 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:42,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:42,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:42,345 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:42,345 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:42,345 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:42,345 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:42,345 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:42,345 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:42,345 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:42,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:42,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:42,345 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:42,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:42,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:42,346 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:42,346 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:42,347 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:56:42,356 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:42,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 493 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:41456 deadline: 1679309862356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:42,447 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:42,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:42,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:42,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:42,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:42,474 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:42,474 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:42,474 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:42,474 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:42,479 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:42,479 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:42,489 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:42,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:42,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:42,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:42,490 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:42,492 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:42,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:42,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:42,493 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:42,494 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:42,494 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:42,494 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:42,494 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:42,494 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:42,494 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:42,494 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:42,502 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:42,502 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:42,503 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:42,503 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. started... 2023-03-20 10:56:42,503 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. as already flushing 2023-03-20 10:56:42,503 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:42,503 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:42,503 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:42,503 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:42,504 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:42,504 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:42,505 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:42,505 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:42,505 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:42,510 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:42,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:42,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:42,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:42,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:42,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:42,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:42,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:42,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:42,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:42,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f509080[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:42,514 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:42,514 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:42,521 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:42,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:42,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:42,521 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:42,521 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:42,521 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:42,522 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:42,522 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:42,527 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:42,527 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:42,527 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:42,530 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:42,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:42,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:42,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:42,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:42,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:42,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:42,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:42,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:42,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:42,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:42,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:42,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:42,562 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:42,568 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:42,568 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:42,568 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:42,568 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:42,575 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:42,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:42,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:42,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:42,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:42,576 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:42,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:42,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:42,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:42,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:42,577 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:42,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:42,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:42,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:42,578 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:42,578 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:42,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:42,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:42,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:42,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:42,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:42,580 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:42,580 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:42,580 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:42,580 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:42,580 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:42,580 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:42,580 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:42,580 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:42,581 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:42,581 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:42,582 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:42,582 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:42,582 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:42,582 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:42,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:42,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:42,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:42,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:42,682 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:42,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:42,683 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=633 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/C/bb1cc6768d244ea99b3e3cb5b342f9b7 2023-03-20 10:56:42,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:42,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:42,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:42,694 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:42,693 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/A/e42c1e789f7448629cc30ef73dbcf918 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/e42c1e789f7448629cc30ef73dbcf918 2023-03-20 10:56:42,694 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:42,694 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:42,694 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:42,698 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:42,698 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:42,699 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:42,699 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:42,699 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:42,702 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:42,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:42,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:42,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:42,703 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:42,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:42,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:42,703 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:42,708 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:42,708 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:42,708 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/e42c1e789f7448629cc30ef73dbcf918, entries=250, sequenceid=633, filesize=16.7 K 2023-03-20 10:56:42,709 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/B/d3f46b5064d743f8a642430ea9e5b067 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/d3f46b5064d743f8a642430ea9e5b067 2023-03-20 10:56:42,709 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:42,710 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:42,710 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:42,710 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:42,710 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:42,710 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:42,710 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. started... 2023-03-20 10:56:42,710 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. as already flushing 2023-03-20 10:56:42,710 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:42,710 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:42,711 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:42,711 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:42,711 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:42,711 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:42,712 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:42,712 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:42,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:42,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:42,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:42,712 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:42,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:42,712 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:42,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:42,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:42,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:42,725 DEBUG [Thread-156] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x38ebb9f9 to 127.0.0.1:60226 2023-03-20 10:56:42,725 DEBUG [Thread-156] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 10:56:42,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:42,727 DEBUG [Thread-154] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5fe050a3 to 127.0.0.1:60226 2023-03-20 10:56:42,727 DEBUG [Thread-154] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 10:56:42,727 DEBUG [Thread-152] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x127a87ef to 127.0.0.1:60226 2023-03-20 10:56:42,727 DEBUG [Thread-152] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 10:56:42,727 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:42,727 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ab270cd[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:42,727 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:42,727 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:42,730 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/d3f46b5064d743f8a642430ea9e5b067, entries=150, sequenceid=633, filesize=11.9 K 2023-03-20 10:56:42,731 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:42,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:42,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:42,731 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:42,731 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:42,731 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:42,731 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/C/bb1cc6768d244ea99b3e3cb5b342f9b7 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/bb1cc6768d244ea99b3e3cb5b342f9b7 2023-03-20 10:56:42,732 DEBUG [Thread-158] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x13112432 to 127.0.0.1:60226 2023-03-20 10:56:42,732 DEBUG [Thread-158] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 10:56:42,740 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/bb1cc6768d244ea99b3e3cb5b342f9b7, entries=150, sequenceid=633, filesize=11.9 K 2023-03-20 10:56:42,741 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~420.80 KB/430896, currentSize=33.54 KB/34350 for c79c35ba58052cacab6709e8682a60f1 in 1091ms, sequenceid=633, compaction requested=true 2023-03-20 10:56:42,741 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c79c35ba58052cacab6709e8682a60f1: 2023-03-20 10:56:42,741 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:56:42,741 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 10:56:42,741 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 10:56:42,741 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:56:42,742 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:56:42,743 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47322 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 10:56:42,743 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 54642 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 10:56:42,743 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): c79c35ba58052cacab6709e8682a60f1/B is initiating minor compaction (all files) 2023-03-20 10:56:42,743 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1912): c79c35ba58052cacab6709e8682a60f1/A is initiating minor compaction (all files) 2023-03-20 10:56:42,743 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c79c35ba58052cacab6709e8682a60f1/B in TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:42,743 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c79c35ba58052cacab6709e8682a60f1/A in TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:42,744 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/375926c82d3545b9b229022aaa924921, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/f7fc420850f14bb5b4fcfc79f6c9c0bf, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/0e2478befe064772a46ce50444e12782, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/d3f46b5064d743f8a642430ea9e5b067] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp, totalSize=46.2 K 2023-03-20 10:56:42,744 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/47a0379ab76f4631a18c34a2d156e704, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/2d8b8661bde740148d10231de04c2bf7, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/fe2b7ae0612a46fead7240fea74c2eea, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/e42c1e789f7448629cc30ef73dbcf918] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp, totalSize=53.4 K 2023-03-20 10:56:42,744 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 375926c82d3545b9b229022aaa924921, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=546, earliestPutTs=1679309797631 2023-03-20 10:56:42,744 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 47a0379ab76f4631a18c34a2d156e704, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=546, earliestPutTs=1679309797631 2023-03-20 10:56:42,744 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 2d8b8661bde740148d10231de04c2bf7, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=582, earliestPutTs=1679309799044 2023-03-20 10:56:42,744 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting f7fc420850f14bb5b4fcfc79f6c9c0bf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=582, earliestPutTs=1679309799044 2023-03-20 10:56:42,745 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting fe2b7ae0612a46fead7240fea74c2eea, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=595, earliestPutTs=1679309799131 2023-03-20 10:56:42,745 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 0e2478befe064772a46ce50444e12782, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=595, earliestPutTs=1679309799131 2023-03-20 10:56:42,745 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting e42c1e789f7448629cc30ef73dbcf918, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=633, earliestPutTs=1679309800442 2023-03-20 10:56:42,746 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting d3f46b5064d743f8a642430ea9e5b067, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=633, earliestPutTs=1679309800442 2023-03-20 10:56:42,765 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] throttle.PressureAwareThroughputController(145): c79c35ba58052cacab6709e8682a60f1#A#compaction#102 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:56:42,770 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): c79c35ba58052cacab6709e8682a60f1#B#compaction#103 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:56:42,787 DEBUG [Thread-145] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x13e837e0 to 127.0.0.1:60226 2023-03-20 10:56:42,787 DEBUG [Thread-145] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 10:56:42,832 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:42,832 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:42,832 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:42,836 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:42,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on c79c35ba58052cacab6709e8682a60f1 2023-03-20 10:56:42,859 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c79c35ba58052cacab6709e8682a60f1 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-20 10:56:42,859 DEBUG [Thread-149] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6be8c20b to 127.0.0.1:60226 2023-03-20 10:56:42,859 DEBUG [Thread-149] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 10:56:42,859 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=A 2023-03-20 10:56:42,859 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:42,859 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=B 2023-03-20 10:56:42,859 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:42,859 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=C 2023-03-20 10:56:42,859 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:42,868 DEBUG [Thread-143] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x20e4e4f3 to 127.0.0.1:60226 2023-03-20 10:56:42,868 DEBUG [Thread-143] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 10:56:42,871 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=643 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/A/875f685a66e14c529d504d3b11f450e4 2023-03-20 10:56:42,884 DEBUG [Thread-141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x05be6782 to 127.0.0.1:60226 2023-03-20 10:56:42,884 DEBUG [Thread-141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 10:56:42,925 DEBUG [Thread-147] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6362e5a0 to 127.0.0.1:60226 2023-03-20 10:56:42,925 DEBUG [Thread-147] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 10:56:42,938 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:42,938 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:42,938 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:42,938 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:42,938 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:42,938 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:42,938 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:43,145 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:43,145 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:43,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:43,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:43,145 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:43,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:43,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:43,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:43,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:43,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:43,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:43,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:43,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:43,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:43,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:43,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:43,150 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:43,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:43,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:43,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:43,151 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:43,161 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:43,161 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:43,161 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:43,161 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:43,161 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:43,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:43,161 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:43,162 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:43,162 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:43,162 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:43,162 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:43,162 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:43,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:43,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:43,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:43,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:43,166 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:43,166 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:43,190 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/A/c7df9dad4eca441cb7fc9254799fb9e1 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/c7df9dad4eca441cb7fc9254799fb9e1 2023-03-20 10:56:43,208 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c79c35ba58052cacab6709e8682a60f1/A of c79c35ba58052cacab6709e8682a60f1 into c7df9dad4eca441cb7fc9254799fb9e1(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:56:43,208 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c79c35ba58052cacab6709e8682a60f1: 2023-03-20 10:56:43,208 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1., storeName=c79c35ba58052cacab6709e8682a60f1/A, priority=12, startTime=1679309802741; duration=0sec 2023-03-20 10:56:43,208 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:56:43,208 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 10:56:43,212 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/B/dd9a39a217a54a4a8dc09e32844b9bce as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/dd9a39a217a54a4a8dc09e32844b9bce 2023-03-20 10:56:43,212 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47322 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 10:56:43,212 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1912): c79c35ba58052cacab6709e8682a60f1/C is initiating minor compaction (all files) 2023-03-20 10:56:43,212 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c79c35ba58052cacab6709e8682a60f1/C in TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:43,212 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/4277964fb0004c95bd2c5e12a20b92f8, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/56488fdb03274d96a17759a6cbfeffe9, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/cd398a19719d45618bf95e294f0bea7b, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/bb1cc6768d244ea99b3e3cb5b342f9b7] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp, totalSize=46.2 K 2023-03-20 10:56:43,212 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 4277964fb0004c95bd2c5e12a20b92f8, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=546, earliestPutTs=1679309797631 2023-03-20 10:56:43,213 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 56488fdb03274d96a17759a6cbfeffe9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=582, earliestPutTs=1679309799044 2023-03-20 10:56:43,213 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting cd398a19719d45618bf95e294f0bea7b, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=595, earliestPutTs=1679309799131 2023-03-20 10:56:43,213 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting bb1cc6768d244ea99b3e3cb5b342f9b7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=633, earliestPutTs=1679309800442 2023-03-20 10:56:43,217 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c79c35ba58052cacab6709e8682a60f1/B of c79c35ba58052cacab6709e8682a60f1 into dd9a39a217a54a4a8dc09e32844b9bce(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:56:43,217 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c79c35ba58052cacab6709e8682a60f1: 2023-03-20 10:56:43,217 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1., storeName=c79c35ba58052cacab6709e8682a60f1/B, priority=12, startTime=1679309802741; duration=0sec 2023-03-20 10:56:43,217 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:56:43,221 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] throttle.PressureAwareThroughputController(145): c79c35ba58052cacab6709e8682a60f1#C#compaction#106 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:56:43,266 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:43,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:43,267 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-20 10:56:43,268 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(394): wrote 30 2023-03-20 10:56:43,268 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(394): wrote 50 2023-03-20 10:56:43,268 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(394): wrote 42 2023-03-20 10:56:43,268 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(394): wrote 49 2023-03-20 10:56:43,268 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(394): wrote 369 2023-03-20 10:56:43,268 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-20 10:56:43,268 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(398): read 2829 2023-03-20 10:56:43,268 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(398): read 2745 2023-03-20 10:56:43,268 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-20 10:56:43,268 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(402): scanned 1334 2023-03-20 10:56:43,268 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(403): verified 4000 rows 2023-03-20 10:56:43,268 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(402): scanned 1332 2023-03-20 10:56:43,268 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(403): verified 3992 rows 2023-03-20 10:56:43,268 INFO [Listener at localhost.localdomain/43509] client.ConnectionImplementation(1959): Closing master protocol: MasterService 2023-03-20 10:56:43,269 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x18660fc4 to 127.0.0.1:60226 2023-03-20 10:56:43,269 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 10:56:43,272 INFO [Listener at localhost.localdomain/43509] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-20 10:56:43,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-03-20 10:56:43,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-20 10:56:43,290 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679309803290"}]},"ts":"1679309803290"} 2023-03-20 10:56:43,292 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-20 10:56:43,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-20 10:56:43,301 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=643 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/B/debcc7fc40ef4d11a3feda9fff6f5364 2023-03-20 10:56:43,373 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-20 10:56:43,375 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c79c35ba58052cacab6709e8682a60f1, UNASSIGN}] 2023-03-20 10:56:43,377 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c79c35ba58052cacab6709e8682a60f1, UNASSIGN 2023-03-20 10:56:43,378 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=c79c35ba58052cacab6709e8682a60f1, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:43,379 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679309803378"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679309803378"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679309803378"}]},"ts":"1679309803378"} 2023-03-20 10:56:43,380 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062}] 2023-03-20 10:56:43,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-20 10:56:43,538 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close c79c35ba58052cacab6709e8682a60f1 2023-03-20 10:56:43,538 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing c79c35ba58052cacab6709e8682a60f1, disabling compactions & flushes 2023-03-20 10:56:43,538 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1858): waiting for 1 compactions & cache flush to complete for region TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:43,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-20 10:56:43,638 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/C/30b3016e95ba4cb69fa22b7e05ba1984 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/30b3016e95ba4cb69fa22b7e05ba1984 2023-03-20 10:56:43,644 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c79c35ba58052cacab6709e8682a60f1/C of c79c35ba58052cacab6709e8682a60f1 into 30b3016e95ba4cb69fa22b7e05ba1984(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:56:43,644 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c79c35ba58052cacab6709e8682a60f1: 2023-03-20 10:56:43,644 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1., storeName=c79c35ba58052cacab6709e8682a60f1/C, priority=12, startTime=1679309802742; duration=0sec 2023-03-20 10:56:43,644 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:43,644 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:56:43,714 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=643 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/C/cbe13239165c43dca6b264346c173c8c 2023-03-20 10:56:43,721 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/A/875f685a66e14c529d504d3b11f450e4 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/875f685a66e14c529d504d3b11f450e4 2023-03-20 10:56:43,726 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/875f685a66e14c529d504d3b11f450e4, entries=100, sequenceid=643, filesize=9.5 K 2023-03-20 10:56:43,729 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/B/debcc7fc40ef4d11a3feda9fff6f5364 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/debcc7fc40ef4d11a3feda9fff6f5364 2023-03-20 10:56:43,735 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/debcc7fc40ef4d11a3feda9fff6f5364, entries=100, sequenceid=643, filesize=9.5 K 2023-03-20 10:56:43,739 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/C/cbe13239165c43dca6b264346c173c8c as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/cbe13239165c43dca6b264346c173c8c 2023-03-20 10:56:43,747 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/cbe13239165c43dca6b264346c173c8c, entries=100, sequenceid=643, filesize=9.5 K 2023-03-20 10:56:43,748 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=20.13 KB/20610 for c79c35ba58052cacab6709e8682a60f1 in 889ms, sequenceid=643, compaction requested=false 2023-03-20 10:56:43,748 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c79c35ba58052cacab6709e8682a60f1: 2023-03-20 10:56:43,749 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:43,749 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:43,749 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. after waiting 0 ms 2023-03-20 10:56:43,749 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:43,749 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing c79c35ba58052cacab6709e8682a60f1 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-03-20 10:56:43,749 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=A 2023-03-20 10:56:43,749 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:43,750 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=B 2023-03-20 10:56:43,750 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:43,750 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c79c35ba58052cacab6709e8682a60f1, store=C 2023-03-20 10:56:43,750 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:43,767 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=652 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/A/04aed902769742919bc135e68a851922 2023-03-20 10:56:43,780 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=652 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/B/e2a4f46ae82f45d2af8d18e8a27ba875 2023-03-20 10:56:43,816 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=652 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/C/d182c2dc43de4f9995e57f96f479b346 2023-03-20 10:56:43,824 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/A/04aed902769742919bc135e68a851922 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/04aed902769742919bc135e68a851922 2023-03-20 10:56:43,830 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/04aed902769742919bc135e68a851922, entries=150, sequenceid=652, filesize=11.9 K 2023-03-20 10:56:43,831 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/B/e2a4f46ae82f45d2af8d18e8a27ba875 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/e2a4f46ae82f45d2af8d18e8a27ba875 2023-03-20 10:56:43,850 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/e2a4f46ae82f45d2af8d18e8a27ba875, entries=150, sequenceid=652, filesize=11.9 K 2023-03-20 10:56:43,851 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/.tmp/C/d182c2dc43de4f9995e57f96f479b346 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/d182c2dc43de4f9995e57f96f479b346 2023-03-20 10:56:43,857 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/d182c2dc43de4f9995e57f96f479b346, entries=150, sequenceid=652, filesize=11.9 K 2023-03-20 10:56:43,858 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for c79c35ba58052cacab6709e8682a60f1 in 109ms, sequenceid=652, compaction requested=true 2023-03-20 10:56:43,896 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/6f8496f473744f93b6ee28aef555c04f, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/52797d30340d48f28cf1b73545a98c45, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/780061fdcd3c4cdebd25ad2b6d906b02, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/ed58a95cba484f9a8b766ded56c5b6f0, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/0744e7335f794d89b78d0b8374b959ae, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/6ee6083833fc437a93190b9610c7ed05, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/9038b849757243ae818620cc35239708, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/ae31770d2a314d17bf700433045ca163, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/59c440e99c19402a835f576e27803102, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/ec1ba99ed55a4b55befc06b7f94cc590, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/024b7158c3394e8baf3d42e4aee5a35a, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/2892c493c58f4e669d30af687e5d4267, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/21f87be3656a4eab8d7d3ee98676a8c6, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/a7b96586d19f411291abab7e85b7e51c, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/edbfd599255848df8334028b263c46a1, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/19e5b29188d243bd9aec3d472c4d1131, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/c709adc09d36496e941d6f522cc07b84, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/76794a01628e4aa5b96ad3cc301cbe32, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/a4a5c5e3266e41e5ba27e45d9dcf6bbf, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/5f913c4c34a54f6e9b8aca5e554e9ca1, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/abb3833bf0d04557a5bee373c8632798, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/c7c518a9f4124023aadf3a7e2a39e3ca, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/c2f5a9522cc745d98a47d7ba697b77a9, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/942c1a5846f14fcc9bc9f5cd62570a29, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/cd3b7cc9b2a24779810b424c3bf72a37, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/4dd6571afaf2452db9dfb64517d8e2dd, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/6a279bc773bf4eb0903089ce5bb7da77, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/5f9bc8713b694de8895d88b3e1aafbaf, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/f138c61dd4bd4c5dbf2d83e1de80bce3, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/47a0379ab76f4631a18c34a2d156e704, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/747eb8fa964a414cbeeb03b3afdf45c9, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/2d8b8661bde740148d10231de04c2bf7, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/fe2b7ae0612a46fead7240fea74c2eea, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/e42c1e789f7448629cc30ef73dbcf918] to archive 2023-03-20 10:56:43,897 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-20 10:56:43,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-20 10:56:43,905 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/6f8496f473744f93b6ee28aef555c04f to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/6f8496f473744f93b6ee28aef555c04f 2023-03-20 10:56:43,907 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/52797d30340d48f28cf1b73545a98c45 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/52797d30340d48f28cf1b73545a98c45 2023-03-20 10:56:43,908 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/780061fdcd3c4cdebd25ad2b6d906b02 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/780061fdcd3c4cdebd25ad2b6d906b02 2023-03-20 10:56:43,912 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/ed58a95cba484f9a8b766ded56c5b6f0 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/ed58a95cba484f9a8b766ded56c5b6f0 2023-03-20 10:56:43,914 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/0744e7335f794d89b78d0b8374b959ae to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/0744e7335f794d89b78d0b8374b959ae 2023-03-20 10:56:43,918 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/6ee6083833fc437a93190b9610c7ed05 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/6ee6083833fc437a93190b9610c7ed05 2023-03-20 10:56:43,922 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/9038b849757243ae818620cc35239708 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/9038b849757243ae818620cc35239708 2023-03-20 10:56:43,924 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/ae31770d2a314d17bf700433045ca163 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/ae31770d2a314d17bf700433045ca163 2023-03-20 10:56:43,926 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/59c440e99c19402a835f576e27803102 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/59c440e99c19402a835f576e27803102 2023-03-20 10:56:43,928 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/ec1ba99ed55a4b55befc06b7f94cc590 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/ec1ba99ed55a4b55befc06b7f94cc590 2023-03-20 10:56:43,930 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/024b7158c3394e8baf3d42e4aee5a35a to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/024b7158c3394e8baf3d42e4aee5a35a 2023-03-20 10:56:43,933 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/2892c493c58f4e669d30af687e5d4267 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/2892c493c58f4e669d30af687e5d4267 2023-03-20 10:56:43,938 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/21f87be3656a4eab8d7d3ee98676a8c6 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/21f87be3656a4eab8d7d3ee98676a8c6 2023-03-20 10:56:43,949 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/a7b96586d19f411291abab7e85b7e51c to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/a7b96586d19f411291abab7e85b7e51c 2023-03-20 10:56:43,952 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/edbfd599255848df8334028b263c46a1 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/edbfd599255848df8334028b263c46a1 2023-03-20 10:56:43,956 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/19e5b29188d243bd9aec3d472c4d1131 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/19e5b29188d243bd9aec3d472c4d1131 2023-03-20 10:56:43,960 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/c709adc09d36496e941d6f522cc07b84 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/c709adc09d36496e941d6f522cc07b84 2023-03-20 10:56:43,967 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/76794a01628e4aa5b96ad3cc301cbe32 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/76794a01628e4aa5b96ad3cc301cbe32 2023-03-20 10:56:43,971 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/a4a5c5e3266e41e5ba27e45d9dcf6bbf to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/a4a5c5e3266e41e5ba27e45d9dcf6bbf 2023-03-20 10:56:43,974 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/5f913c4c34a54f6e9b8aca5e554e9ca1 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/5f913c4c34a54f6e9b8aca5e554e9ca1 2023-03-20 10:56:43,976 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/abb3833bf0d04557a5bee373c8632798 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/abb3833bf0d04557a5bee373c8632798 2023-03-20 10:56:43,977 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/c7c518a9f4124023aadf3a7e2a39e3ca to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/c7c518a9f4124023aadf3a7e2a39e3ca 2023-03-20 10:56:43,978 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/c2f5a9522cc745d98a47d7ba697b77a9 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/c2f5a9522cc745d98a47d7ba697b77a9 2023-03-20 10:56:43,979 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/942c1a5846f14fcc9bc9f5cd62570a29 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/942c1a5846f14fcc9bc9f5cd62570a29 2023-03-20 10:56:43,981 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/cd3b7cc9b2a24779810b424c3bf72a37 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/cd3b7cc9b2a24779810b424c3bf72a37 2023-03-20 10:56:43,982 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/4dd6571afaf2452db9dfb64517d8e2dd to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/4dd6571afaf2452db9dfb64517d8e2dd 2023-03-20 10:56:43,983 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/6a279bc773bf4eb0903089ce5bb7da77 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/6a279bc773bf4eb0903089ce5bb7da77 2023-03-20 10:56:43,985 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/5f9bc8713b694de8895d88b3e1aafbaf to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/5f9bc8713b694de8895d88b3e1aafbaf 2023-03-20 10:56:43,986 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/f138c61dd4bd4c5dbf2d83e1de80bce3 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/f138c61dd4bd4c5dbf2d83e1de80bce3 2023-03-20 10:56:43,988 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/47a0379ab76f4631a18c34a2d156e704 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/47a0379ab76f4631a18c34a2d156e704 2023-03-20 10:56:43,989 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/747eb8fa964a414cbeeb03b3afdf45c9 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/747eb8fa964a414cbeeb03b3afdf45c9 2023-03-20 10:56:43,991 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/2d8b8661bde740148d10231de04c2bf7 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/2d8b8661bde740148d10231de04c2bf7 2023-03-20 10:56:43,993 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/fe2b7ae0612a46fead7240fea74c2eea to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/fe2b7ae0612a46fead7240fea74c2eea 2023-03-20 10:56:43,997 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/e42c1e789f7448629cc30ef73dbcf918 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/e42c1e789f7448629cc30ef73dbcf918 2023-03-20 10:56:44,029 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/f9d07964968a4977a1c53e7fc59cb66b, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/44209885eda1465390efd062337c353e, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/ff4926af8be244959805131528248237, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/eb6f0d88534d474f87a0af1e88c0a639, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/6b2989c71b9d492ea297bfebf920c659, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/b02e7361f3374526b36dcdbcd3ff84d4, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/927f030c43e74efd99f74406ebf0ca71, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/226c26459a444a229fa338b447e9d2b9, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/f3b4ffa0081646978549f39b6c2429f3, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/3532c85c18de4153ba976b48a0f75655, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/45246ae816c24193a8df207b20a98452, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/4f521188449745e0b8fbbf0279406775, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/7476868f0b70473fa287eead7e80a66b, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/04a0d2f7ceaf48ddac851170d168e83c, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/cdeedb8f493848b0a6d4169dfda1f961, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/72c76ad14a7140269a2173d3f6ad3180, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/ff7e43113ce64683a932a0811942de17, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/682afc884e684f7c8794383ebb8cdbb1, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/3a3ce55963f24bb78b7ec45526c2d082, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/8a503bcb43d747a5b70c610fd865e1b6, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/51cf546658c74aba849b6aece7774cf7, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/ae791a0044fb4bb99372369d5005a959, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/2b0dde7e4974450c99f04e9867a2ffd3, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/c6ff30ad3b8b44dcbc17365c40828c44, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/5838a70c16524ac49b30fd9a085bde64, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/ab28eaf508db4f5ca916dc619f2b74c5, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/50eee295b73440a1ab45e8c57ce15d31, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/640150bfc6f64975b37c7ee0f725624a, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/313c935cf3774993baeef8013222033b, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/375926c82d3545b9b229022aaa924921, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/cf75be2f3a904225b3477df831fd3381, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/f7fc420850f14bb5b4fcfc79f6c9c0bf, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/0e2478befe064772a46ce50444e12782, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/d3f46b5064d743f8a642430ea9e5b067] to archive 2023-03-20 10:56:44,030 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-20 10:56:44,032 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/f9d07964968a4977a1c53e7fc59cb66b to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/f9d07964968a4977a1c53e7fc59cb66b 2023-03-20 10:56:44,034 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/44209885eda1465390efd062337c353e to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/44209885eda1465390efd062337c353e 2023-03-20 10:56:44,036 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/ff4926af8be244959805131528248237 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/ff4926af8be244959805131528248237 2023-03-20 10:56:44,038 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/eb6f0d88534d474f87a0af1e88c0a639 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/eb6f0d88534d474f87a0af1e88c0a639 2023-03-20 10:56:44,040 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/6b2989c71b9d492ea297bfebf920c659 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/6b2989c71b9d492ea297bfebf920c659 2023-03-20 10:56:44,042 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/b02e7361f3374526b36dcdbcd3ff84d4 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/b02e7361f3374526b36dcdbcd3ff84d4 2023-03-20 10:56:44,043 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/927f030c43e74efd99f74406ebf0ca71 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/927f030c43e74efd99f74406ebf0ca71 2023-03-20 10:56:44,045 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/226c26459a444a229fa338b447e9d2b9 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/226c26459a444a229fa338b447e9d2b9 2023-03-20 10:56:44,046 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/f3b4ffa0081646978549f39b6c2429f3 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/f3b4ffa0081646978549f39b6c2429f3 2023-03-20 10:56:44,048 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/3532c85c18de4153ba976b48a0f75655 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/3532c85c18de4153ba976b48a0f75655 2023-03-20 10:56:44,049 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/45246ae816c24193a8df207b20a98452 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/45246ae816c24193a8df207b20a98452 2023-03-20 10:56:44,051 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/4f521188449745e0b8fbbf0279406775 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/4f521188449745e0b8fbbf0279406775 2023-03-20 10:56:44,052 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/7476868f0b70473fa287eead7e80a66b to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/7476868f0b70473fa287eead7e80a66b 2023-03-20 10:56:44,053 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/04a0d2f7ceaf48ddac851170d168e83c to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/04a0d2f7ceaf48ddac851170d168e83c 2023-03-20 10:56:44,054 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/cdeedb8f493848b0a6d4169dfda1f961 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/cdeedb8f493848b0a6d4169dfda1f961 2023-03-20 10:56:44,055 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/72c76ad14a7140269a2173d3f6ad3180 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/72c76ad14a7140269a2173d3f6ad3180 2023-03-20 10:56:44,056 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/ff7e43113ce64683a932a0811942de17 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/ff7e43113ce64683a932a0811942de17 2023-03-20 10:56:44,057 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/682afc884e684f7c8794383ebb8cdbb1 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/682afc884e684f7c8794383ebb8cdbb1 2023-03-20 10:56:44,058 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/3a3ce55963f24bb78b7ec45526c2d082 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/3a3ce55963f24bb78b7ec45526c2d082 2023-03-20 10:56:44,060 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/8a503bcb43d747a5b70c610fd865e1b6 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/8a503bcb43d747a5b70c610fd865e1b6 2023-03-20 10:56:44,062 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/51cf546658c74aba849b6aece7774cf7 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/51cf546658c74aba849b6aece7774cf7 2023-03-20 10:56:44,063 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/ae791a0044fb4bb99372369d5005a959 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/ae791a0044fb4bb99372369d5005a959 2023-03-20 10:56:44,065 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/2b0dde7e4974450c99f04e9867a2ffd3 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/2b0dde7e4974450c99f04e9867a2ffd3 2023-03-20 10:56:44,068 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/c6ff30ad3b8b44dcbc17365c40828c44 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/c6ff30ad3b8b44dcbc17365c40828c44 2023-03-20 10:56:44,070 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/5838a70c16524ac49b30fd9a085bde64 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/5838a70c16524ac49b30fd9a085bde64 2023-03-20 10:56:44,071 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/ab28eaf508db4f5ca916dc619f2b74c5 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/ab28eaf508db4f5ca916dc619f2b74c5 2023-03-20 10:56:44,074 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/50eee295b73440a1ab45e8c57ce15d31 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/50eee295b73440a1ab45e8c57ce15d31 2023-03-20 10:56:44,075 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/640150bfc6f64975b37c7ee0f725624a to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/640150bfc6f64975b37c7ee0f725624a 2023-03-20 10:56:44,077 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/313c935cf3774993baeef8013222033b to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/313c935cf3774993baeef8013222033b 2023-03-20 10:56:44,078 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/375926c82d3545b9b229022aaa924921 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/375926c82d3545b9b229022aaa924921 2023-03-20 10:56:44,080 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/cf75be2f3a904225b3477df831fd3381 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/cf75be2f3a904225b3477df831fd3381 2023-03-20 10:56:44,081 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/f7fc420850f14bb5b4fcfc79f6c9c0bf to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/f7fc420850f14bb5b4fcfc79f6c9c0bf 2023-03-20 10:56:44,082 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/0e2478befe064772a46ce50444e12782 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/0e2478befe064772a46ce50444e12782 2023-03-20 10:56:44,084 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/d3f46b5064d743f8a642430ea9e5b067 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/d3f46b5064d743f8a642430ea9e5b067 2023-03-20 10:56:44,089 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/1ace291dac7c4e52ac81c456d1c2ef03, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/1f5f897a388c4cc4851260140369eb2f, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/863198e70a3d4dca9ec1055e36d1dcd2, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/e8cf37fb88d148c1b0d8095add427b70, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/58b5d8bffcd64553afcd411518ae5c63, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/d9e2ed4b3ac94996b6f925eccc55bcec, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/4fbc60dd9f3b4654b8f9ca1ac325a7e2, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/23ba5c831bbc4eddbe9dce7adc592ee2, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/4101d396c7f044c7ae6713a99c68c8b4, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/6f92b70c99b2459891ef598cda744076, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/c3125f648e7349dc958d0afb0ee8ff78, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/07435a7fb05a4f9ca5f789352a4695ae, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/b6490a7fd52c4005b6c607d9a300f7d5, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/1e0b26e416cb48deb3f9c94376e47a8c, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/12919e9f7e1e4b91904908a980b56be4, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/75e6c15dee4c40748e632457edd1e209, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/977f0a046b504392896f1de49b3d7d6f, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/e9b8cc68bbc1490bb6f47317a6e320a3, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/d4f2e38940864b24bf3c1cc838570cb7, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/3b338e20b5504ab1bf74a771fa854ee7, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/5752a8ac55a14723bb95453c29d4139e, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/9115a29c7faf4f7390d6fec9dbb2167b, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/a1181ab928014935aafea6b32f9cf36b, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/d4c5b134217f4b00b949766cce010ecd, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/b1ade1a10b5e4669bf491d676c5cffb4, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/332b2db836f44ec19a3c283936198e1d, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/4e75bb1db9f9459580b2034fb15ad88b, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/7463bc21680c4c898d85662db3a8d0b5, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/829d43168bf349c1adece151d011b160, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/4277964fb0004c95bd2c5e12a20b92f8, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/514fbc06db584f37bd70e3ef4a6292fc, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/56488fdb03274d96a17759a6cbfeffe9, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/cd398a19719d45618bf95e294f0bea7b, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/bb1cc6768d244ea99b3e3cb5b342f9b7] to archive 2023-03-20 10:56:44,090 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-20 10:56:44,094 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/1ace291dac7c4e52ac81c456d1c2ef03 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/1ace291dac7c4e52ac81c456d1c2ef03 2023-03-20 10:56:44,095 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/1f5f897a388c4cc4851260140369eb2f to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/1f5f897a388c4cc4851260140369eb2f 2023-03-20 10:56:44,096 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/863198e70a3d4dca9ec1055e36d1dcd2 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/863198e70a3d4dca9ec1055e36d1dcd2 2023-03-20 10:56:44,104 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/e8cf37fb88d148c1b0d8095add427b70 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/e8cf37fb88d148c1b0d8095add427b70 2023-03-20 10:56:44,106 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/58b5d8bffcd64553afcd411518ae5c63 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/58b5d8bffcd64553afcd411518ae5c63 2023-03-20 10:56:44,107 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/d9e2ed4b3ac94996b6f925eccc55bcec to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/d9e2ed4b3ac94996b6f925eccc55bcec 2023-03-20 10:56:44,109 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/4fbc60dd9f3b4654b8f9ca1ac325a7e2 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/4fbc60dd9f3b4654b8f9ca1ac325a7e2 2023-03-20 10:56:44,116 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/23ba5c831bbc4eddbe9dce7adc592ee2 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/23ba5c831bbc4eddbe9dce7adc592ee2 2023-03-20 10:56:44,117 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/4101d396c7f044c7ae6713a99c68c8b4 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/4101d396c7f044c7ae6713a99c68c8b4 2023-03-20 10:56:44,118 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/6f92b70c99b2459891ef598cda744076 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/6f92b70c99b2459891ef598cda744076 2023-03-20 10:56:44,119 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/c3125f648e7349dc958d0afb0ee8ff78 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/c3125f648e7349dc958d0afb0ee8ff78 2023-03-20 10:56:44,120 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/07435a7fb05a4f9ca5f789352a4695ae to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/07435a7fb05a4f9ca5f789352a4695ae 2023-03-20 10:56:44,121 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/b6490a7fd52c4005b6c607d9a300f7d5 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/b6490a7fd52c4005b6c607d9a300f7d5 2023-03-20 10:56:44,122 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/1e0b26e416cb48deb3f9c94376e47a8c to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/1e0b26e416cb48deb3f9c94376e47a8c 2023-03-20 10:56:44,123 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/12919e9f7e1e4b91904908a980b56be4 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/12919e9f7e1e4b91904908a980b56be4 2023-03-20 10:56:44,124 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/75e6c15dee4c40748e632457edd1e209 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/75e6c15dee4c40748e632457edd1e209 2023-03-20 10:56:44,126 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/977f0a046b504392896f1de49b3d7d6f to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/977f0a046b504392896f1de49b3d7d6f 2023-03-20 10:56:44,127 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/e9b8cc68bbc1490bb6f47317a6e320a3 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/e9b8cc68bbc1490bb6f47317a6e320a3 2023-03-20 10:56:44,129 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/d4f2e38940864b24bf3c1cc838570cb7 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/d4f2e38940864b24bf3c1cc838570cb7 2023-03-20 10:56:44,131 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/3b338e20b5504ab1bf74a771fa854ee7 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/3b338e20b5504ab1bf74a771fa854ee7 2023-03-20 10:56:44,132 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/5752a8ac55a14723bb95453c29d4139e to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/5752a8ac55a14723bb95453c29d4139e 2023-03-20 10:56:44,133 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/9115a29c7faf4f7390d6fec9dbb2167b to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/9115a29c7faf4f7390d6fec9dbb2167b 2023-03-20 10:56:44,135 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/a1181ab928014935aafea6b32f9cf36b to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/a1181ab928014935aafea6b32f9cf36b 2023-03-20 10:56:44,136 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/d4c5b134217f4b00b949766cce010ecd to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/d4c5b134217f4b00b949766cce010ecd 2023-03-20 10:56:44,137 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/b1ade1a10b5e4669bf491d676c5cffb4 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/b1ade1a10b5e4669bf491d676c5cffb4 2023-03-20 10:56:44,139 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/332b2db836f44ec19a3c283936198e1d to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/332b2db836f44ec19a3c283936198e1d 2023-03-20 10:56:44,141 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/4e75bb1db9f9459580b2034fb15ad88b to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/4e75bb1db9f9459580b2034fb15ad88b 2023-03-20 10:56:44,144 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/7463bc21680c4c898d85662db3a8d0b5 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/7463bc21680c4c898d85662db3a8d0b5 2023-03-20 10:56:44,148 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/829d43168bf349c1adece151d011b160 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/829d43168bf349c1adece151d011b160 2023-03-20 10:56:44,155 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/4277964fb0004c95bd2c5e12a20b92f8 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/4277964fb0004c95bd2c5e12a20b92f8 2023-03-20 10:56:44,158 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/514fbc06db584f37bd70e3ef4a6292fc to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/514fbc06db584f37bd70e3ef4a6292fc 2023-03-20 10:56:44,166 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/56488fdb03274d96a17759a6cbfeffe9 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/56488fdb03274d96a17759a6cbfeffe9 2023-03-20 10:56:44,171 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/cd398a19719d45618bf95e294f0bea7b to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/cd398a19719d45618bf95e294f0bea7b 2023-03-20 10:56:44,173 DEBUG [StoreCloser-TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/bb1cc6768d244ea99b3e3cb5b342f9b7 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/bb1cc6768d244ea99b3e3cb5b342f9b7 2023-03-20 10:56:44,178 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/recovered.edits/655.seqid, newMaxSeqId=655, maxSeqId=1 2023-03-20 10:56:44,180 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1. 2023-03-20 10:56:44,180 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for c79c35ba58052cacab6709e8682a60f1: 2023-03-20 10:56:44,182 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed c79c35ba58052cacab6709e8682a60f1 2023-03-20 10:56:44,183 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=c79c35ba58052cacab6709e8682a60f1, regionState=CLOSED 2023-03-20 10:56:44,183 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679309804183"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679309804183"}]},"ts":"1679309804183"} 2023-03-20 10:56:44,186 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-03-20 10:56:44,186 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure c79c35ba58052cacab6709e8682a60f1, server=jenkins-hbase20.apache.org,36041,1679309777062 in 804 msec 2023-03-20 10:56:44,192 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-03-20 10:56:44,192 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=c79c35ba58052cacab6709e8682a60f1, UNASSIGN in 811 msec 2023-03-20 10:56:44,193 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679309804193"}]},"ts":"1679309804193"} 2023-03-20 10:56:44,194 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-20 10:56:44,195 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-20 10:56:44,198 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 917 msec 2023-03-20 10:56:44,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-20 10:56:44,405 INFO [Listener at localhost.localdomain/43509] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 12 completed 2023-03-20 10:56:44,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-03-20 10:56:44,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-20 10:56:44,415 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-20 10:56:44,416 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-20 10:56:44,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-03-20 10:56:44,422 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1 2023-03-20 10:56:44,425 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A, FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B, FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C, FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/recovered.edits] 2023-03-20 10:56:44,428 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/04aed902769742919bc135e68a851922 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/04aed902769742919bc135e68a851922 2023-03-20 10:56:44,431 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/875f685a66e14c529d504d3b11f450e4 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/875f685a66e14c529d504d3b11f450e4 2023-03-20 10:56:44,433 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/c7df9dad4eca441cb7fc9254799fb9e1 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/A/c7df9dad4eca441cb7fc9254799fb9e1 2023-03-20 10:56:44,435 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/dd9a39a217a54a4a8dc09e32844b9bce to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/dd9a39a217a54a4a8dc09e32844b9bce 2023-03-20 10:56:44,437 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/debcc7fc40ef4d11a3feda9fff6f5364 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/debcc7fc40ef4d11a3feda9fff6f5364 2023-03-20 10:56:44,438 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/e2a4f46ae82f45d2af8d18e8a27ba875 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/B/e2a4f46ae82f45d2af8d18e8a27ba875 2023-03-20 10:56:44,441 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/30b3016e95ba4cb69fa22b7e05ba1984 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/30b3016e95ba4cb69fa22b7e05ba1984 2023-03-20 10:56:44,444 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/cbe13239165c43dca6b264346c173c8c to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/cbe13239165c43dca6b264346c173c8c 2023-03-20 10:56:44,447 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/d182c2dc43de4f9995e57f96f479b346 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/C/d182c2dc43de4f9995e57f96f479b346 2023-03-20 10:56:44,450 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/recovered.edits/655.seqid to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1/recovered.edits/655.seqid 2023-03-20 10:56:44,451 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/c79c35ba58052cacab6709e8682a60f1 2023-03-20 10:56:44,451 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-20 10:56:44,454 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-20 10:56:44,475 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-20 10:56:44,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-03-20 10:56:44,522 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-20 10:56:44,524 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-20 10:56:44,524 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-20 10:56:44,524 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1679309804524"}]},"ts":"9223372036854775807"} 2023-03-20 10:56:44,527 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-20 10:56:44,527 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => c79c35ba58052cacab6709e8682a60f1, NAME => 'TestAcidGuarantees,,1679309781402.c79c35ba58052cacab6709e8682a60f1.', STARTKEY => '', ENDKEY => ''}] 2023-03-20 10:56:44,527 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-20 10:56:44,527 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1679309804527"}]},"ts":"9223372036854775807"} 2023-03-20 10:56:44,529 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-20 10:56:44,531 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-20 10:56:44,532 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 122 msec 2023-03-20 10:56:44,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-03-20 10:56:44,719 INFO [Listener at localhost.localdomain/43509] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 15 completed 2023-03-20 10:56:44,733 INFO [Listener at localhost.localdomain/43509] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=238 (was 205) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x663d22de-shared-pool-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-590168113_15 at /127.0.0.1:45036 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-590168113_15 at /127.0.0.1:43932 [Waiting for operation #22] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x663d22de-shared-pool-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: (jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x663d22de-shared-pool-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:0;jenkins-hbase20:36041-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x663d22de-shared-pool-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x663d22de-shared-pool-10 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-590168113_15 at /127.0.0.1:44934 [Waiting for operation #8] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x663d22de-shared-pool-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-590168113_15 at /127.0.0.1:44838 [Waiting for operation #9] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-590168113_15 at /127.0.0.1:44984 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) - Thread LEAK? -, OpenFileDescriptor=582 (was 470) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=709 (was 417) - SystemLoadAverage LEAK? -, ProcessCount=184 (was 184), AvailableMemoryMB=9593 (was 9684) 2023-03-20 10:56:44,745 INFO [Listener at localhost.localdomain/43509] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=238, OpenFileDescriptor=582, MaxFileDescriptor=60000, SystemLoadAverage=709, ProcessCount=184, AvailableMemoryMB=9590 2023-03-20 10:56:44,748 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-20 10:56:44,748 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-20 10:56:44,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure2.ProcedureExecutor(1029): Stored pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-20 10:56:44,755 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-20 10:56:44,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 16 2023-03-20 10:56:44,757 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-20 10:56:44,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-20 10:56:44,760 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded 2023-03-20 10:56:44,760 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded empty. 2023-03-20 10:56:44,761 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded 2023-03-20 10:56:44,761 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-20 10:56:44,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-20 10:56:45,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-20 10:56:45,180 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-20 10:56:45,181 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 9fc4d04a6b2183ce1217c4847d39aded, NAME => 'TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp 2023-03-20 10:56:45,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-20 10:56:45,591 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-20 10:56:45,591 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 9fc4d04a6b2183ce1217c4847d39aded, disabling compactions & flushes 2023-03-20 10:56:45,591 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:45,591 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:45,591 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. after waiting 0 ms 2023-03-20 10:56:45,591 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:45,591 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:45,591 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 9fc4d04a6b2183ce1217c4847d39aded: 2023-03-20 10:56:45,594 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-20 10:56:45,595 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679309805594"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679309805594"}]},"ts":"1679309805594"} 2023-03-20 10:56:45,596 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-20 10:56:45,597 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-20 10:56:45,598 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679309805597"}]},"ts":"1679309805597"} 2023-03-20 10:56:45,599 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-20 10:56:45,601 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=9fc4d04a6b2183ce1217c4847d39aded, ASSIGN}] 2023-03-20 10:56:45,607 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=9fc4d04a6b2183ce1217c4847d39aded, ASSIGN 2023-03-20 10:56:45,608 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=9fc4d04a6b2183ce1217c4847d39aded, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,36041,1679309777062; forceNewPlan=false, retain=false 2023-03-20 10:56:45,759 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=9fc4d04a6b2183ce1217c4847d39aded, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:45,760 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679309805759"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679309805759"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679309805759"}]},"ts":"1679309805759"} 2023-03-20 10:56:45,761 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; OpenRegionProcedure 9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062}] 2023-03-20 10:56:45,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-20 10:56:45,922 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:45,922 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 9fc4d04a6b2183ce1217c4847d39aded, NAME => 'TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.', STARTKEY => '', ENDKEY => ''} 2023-03-20 10:56:45,923 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 9fc4d04a6b2183ce1217c4847d39aded 2023-03-20 10:56:45,923 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-20 10:56:45,923 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 9fc4d04a6b2183ce1217c4847d39aded 2023-03-20 10:56:45,923 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 9fc4d04a6b2183ce1217c4847d39aded 2023-03-20 10:56:45,929 INFO [StoreOpener-9fc4d04a6b2183ce1217c4847d39aded-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 9fc4d04a6b2183ce1217c4847d39aded 2023-03-20 10:56:45,934 DEBUG [StoreOpener-9fc4d04a6b2183ce1217c4847d39aded-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A 2023-03-20 10:56:45,934 DEBUG [StoreOpener-9fc4d04a6b2183ce1217c4847d39aded-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A 2023-03-20 10:56:45,935 INFO [StoreOpener-9fc4d04a6b2183ce1217c4847d39aded-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-20 10:56:45,935 INFO [StoreOpener-9fc4d04a6b2183ce1217c4847d39aded-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9fc4d04a6b2183ce1217c4847d39aded columnFamilyName A 2023-03-20 10:56:45,936 INFO [StoreOpener-9fc4d04a6b2183ce1217c4847d39aded-1] regionserver.HStore(310): Store=9fc4d04a6b2183ce1217c4847d39aded/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-20 10:56:45,936 INFO [StoreOpener-9fc4d04a6b2183ce1217c4847d39aded-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 9fc4d04a6b2183ce1217c4847d39aded 2023-03-20 10:56:45,939 DEBUG [StoreOpener-9fc4d04a6b2183ce1217c4847d39aded-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B 2023-03-20 10:56:45,939 DEBUG [StoreOpener-9fc4d04a6b2183ce1217c4847d39aded-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B 2023-03-20 10:56:45,940 INFO [StoreOpener-9fc4d04a6b2183ce1217c4847d39aded-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-20 10:56:45,940 INFO [StoreOpener-9fc4d04a6b2183ce1217c4847d39aded-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9fc4d04a6b2183ce1217c4847d39aded columnFamilyName B 2023-03-20 10:56:45,941 INFO [StoreOpener-9fc4d04a6b2183ce1217c4847d39aded-1] regionserver.HStore(310): Store=9fc4d04a6b2183ce1217c4847d39aded/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-20 10:56:45,941 INFO [StoreOpener-9fc4d04a6b2183ce1217c4847d39aded-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 9fc4d04a6b2183ce1217c4847d39aded 2023-03-20 10:56:45,943 DEBUG [StoreOpener-9fc4d04a6b2183ce1217c4847d39aded-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C 2023-03-20 10:56:45,943 DEBUG [StoreOpener-9fc4d04a6b2183ce1217c4847d39aded-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C 2023-03-20 10:56:45,943 INFO [StoreOpener-9fc4d04a6b2183ce1217c4847d39aded-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-20 10:56:45,943 INFO [StoreOpener-9fc4d04a6b2183ce1217c4847d39aded-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9fc4d04a6b2183ce1217c4847d39aded columnFamilyName C 2023-03-20 10:56:45,944 INFO [StoreOpener-9fc4d04a6b2183ce1217c4847d39aded-1] regionserver.HStore(310): Store=9fc4d04a6b2183ce1217c4847d39aded/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-20 10:56:45,944 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:45,945 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded 2023-03-20 10:56:45,945 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded 2023-03-20 10:56:45,947 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-20 10:56:45,948 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 9fc4d04a6b2183ce1217c4847d39aded 2023-03-20 10:56:45,951 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-20 10:56:45,951 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 9fc4d04a6b2183ce1217c4847d39aded; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63550399, jitterRate=-0.0530252605676651}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-20 10:56:45,951 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 9fc4d04a6b2183ce1217c4847d39aded: 2023-03-20 10:56:45,952 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded., pid=18, masterSystemTime=1679309805913 2023-03-20 10:56:45,955 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:45,955 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:45,955 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=9fc4d04a6b2183ce1217c4847d39aded, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:45,955 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679309805955"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1679309805955"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679309805955"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679309805955"}]},"ts":"1679309805955"} 2023-03-20 10:56:45,959 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-03-20 10:56:45,960 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; OpenRegionProcedure 9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 in 196 msec 2023-03-20 10:56:45,962 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-03-20 10:56:45,962 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=9fc4d04a6b2183ce1217c4847d39aded, ASSIGN in 359 msec 2023-03-20 10:56:45,963 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-20 10:56:45,963 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679309805963"}]},"ts":"1679309805963"} 2023-03-20 10:56:45,965 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-20 10:56:45,969 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-20 10:56:45,973 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=16, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.2220 sec 2023-03-20 10:56:46,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-20 10:56:46,863 INFO [Listener at localhost.localdomain/43509] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 16 completed 2023-03-20 10:56:46,864 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(139): Connect 0x65f279d2 to 127.0.0.1:60226 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 10:56:46,868 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@556231fa, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 10:56:46,869 DEBUG [Listener at localhost.localdomain/43509] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 10:56:46,874 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:42922, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 10:56:46,878 DEBUG [Listener at localhost.localdomain/43509] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-20 10:56:46,881 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:36300, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-20 10:56:46,892 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-20 10:56:46,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.HMaster$12(2454): Client=jenkins//148.251.75.209 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-20 10:56:46,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-03-20 10:56:47,322 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-20 10:56:47,323 DEBUG [PEWorker-2] util.FSTableDescriptors(519): Deleted hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-20 10:56:47,323 INFO [PEWorker-2] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-20 10:56:47,329 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-03-20 10:56:47,342 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=9fc4d04a6b2183ce1217c4847d39aded, REOPEN/MOVE}] 2023-03-20 10:56:47,343 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=9fc4d04a6b2183ce1217c4847d39aded, REOPEN/MOVE 2023-03-20 10:56:47,344 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=9fc4d04a6b2183ce1217c4847d39aded, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:47,344 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679309807344"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679309807344"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679309807344"}]},"ts":"1679309807344"} 2023-03-20 10:56:47,345 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; CloseRegionProcedure 9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062}] 2023-03-20 10:56:47,497 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 9fc4d04a6b2183ce1217c4847d39aded 2023-03-20 10:56:47,498 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 9fc4d04a6b2183ce1217c4847d39aded, disabling compactions & flushes 2023-03-20 10:56:47,498 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:47,498 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:47,498 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. after waiting 0 ms 2023-03-20 10:56:47,498 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:47,501 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-03-20 10:56:47,502 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:47,502 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 9fc4d04a6b2183ce1217c4847d39aded: 2023-03-20 10:56:47,502 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 9fc4d04a6b2183ce1217c4847d39aded to self. 2023-03-20 10:56:47,504 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 9fc4d04a6b2183ce1217c4847d39aded 2023-03-20 10:56:47,504 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=9fc4d04a6b2183ce1217c4847d39aded, regionState=CLOSED 2023-03-20 10:56:47,504 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679309807504"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679309807504"}]},"ts":"1679309807504"} 2023-03-20 10:56:47,508 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-03-20 10:56:47,508 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; CloseRegionProcedure 9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 in 161 msec 2023-03-20 10:56:47,508 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=9fc4d04a6b2183ce1217c4847d39aded, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase20.apache.org,36041,1679309777062; forceNewPlan=false, retain=true 2023-03-20 10:56:47,659 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=9fc4d04a6b2183ce1217c4847d39aded, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:47,659 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679309807659"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679309807659"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679309807659"}]},"ts":"1679309807659"} 2023-03-20 10:56:47,661 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=21, state=RUNNABLE; OpenRegionProcedure 9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062}] 2023-03-20 10:56:47,817 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:47,817 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 9fc4d04a6b2183ce1217c4847d39aded, NAME => 'TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.', STARTKEY => '', ENDKEY => ''} 2023-03-20 10:56:47,818 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 9fc4d04a6b2183ce1217c4847d39aded 2023-03-20 10:56:47,818 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-20 10:56:47,818 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 9fc4d04a6b2183ce1217c4847d39aded 2023-03-20 10:56:47,818 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 9fc4d04a6b2183ce1217c4847d39aded 2023-03-20 10:56:47,823 INFO [StoreOpener-9fc4d04a6b2183ce1217c4847d39aded-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 9fc4d04a6b2183ce1217c4847d39aded 2023-03-20 10:56:47,824 DEBUG [StoreOpener-9fc4d04a6b2183ce1217c4847d39aded-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A 2023-03-20 10:56:47,824 DEBUG [StoreOpener-9fc4d04a6b2183ce1217c4847d39aded-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A 2023-03-20 10:56:47,825 INFO [StoreOpener-9fc4d04a6b2183ce1217c4847d39aded-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-20 10:56:47,828 INFO [StoreOpener-9fc4d04a6b2183ce1217c4847d39aded-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9fc4d04a6b2183ce1217c4847d39aded columnFamilyName A 2023-03-20 10:56:47,830 INFO [StoreOpener-9fc4d04a6b2183ce1217c4847d39aded-1] regionserver.HStore(310): Store=9fc4d04a6b2183ce1217c4847d39aded/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-20 10:56:47,831 INFO [StoreOpener-9fc4d04a6b2183ce1217c4847d39aded-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 9fc4d04a6b2183ce1217c4847d39aded 2023-03-20 10:56:47,832 DEBUG [StoreOpener-9fc4d04a6b2183ce1217c4847d39aded-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B 2023-03-20 10:56:47,832 DEBUG [StoreOpener-9fc4d04a6b2183ce1217c4847d39aded-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B 2023-03-20 10:56:47,832 INFO [StoreOpener-9fc4d04a6b2183ce1217c4847d39aded-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-20 10:56:47,833 INFO [StoreOpener-9fc4d04a6b2183ce1217c4847d39aded-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9fc4d04a6b2183ce1217c4847d39aded columnFamilyName B 2023-03-20 10:56:47,833 INFO [StoreOpener-9fc4d04a6b2183ce1217c4847d39aded-1] regionserver.HStore(310): Store=9fc4d04a6b2183ce1217c4847d39aded/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-20 10:56:47,833 INFO [StoreOpener-9fc4d04a6b2183ce1217c4847d39aded-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 9fc4d04a6b2183ce1217c4847d39aded 2023-03-20 10:56:47,834 DEBUG [StoreOpener-9fc4d04a6b2183ce1217c4847d39aded-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C 2023-03-20 10:56:47,834 DEBUG [StoreOpener-9fc4d04a6b2183ce1217c4847d39aded-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C 2023-03-20 10:56:47,834 INFO [StoreOpener-9fc4d04a6b2183ce1217c4847d39aded-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-20 10:56:47,835 INFO [StoreOpener-9fc4d04a6b2183ce1217c4847d39aded-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9fc4d04a6b2183ce1217c4847d39aded columnFamilyName C 2023-03-20 10:56:47,835 INFO [StoreOpener-9fc4d04a6b2183ce1217c4847d39aded-1] regionserver.HStore(310): Store=9fc4d04a6b2183ce1217c4847d39aded/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-20 10:56:47,835 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:47,836 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded 2023-03-20 10:56:47,837 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded 2023-03-20 10:56:47,839 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-20 10:56:47,840 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 9fc4d04a6b2183ce1217c4847d39aded 2023-03-20 10:56:47,841 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 9fc4d04a6b2183ce1217c4847d39aded; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70534399, jitterRate=0.05104444921016693}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-20 10:56:47,841 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 9fc4d04a6b2183ce1217c4847d39aded: 2023-03-20 10:56:47,842 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded., pid=23, masterSystemTime=1679309807812 2023-03-20 10:56:47,843 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:47,843 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:47,844 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=9fc4d04a6b2183ce1217c4847d39aded, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:47,844 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679309807844"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1679309807844"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679309807844"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679309807844"}]},"ts":"1679309807844"} 2023-03-20 10:56:47,847 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=21 2023-03-20 10:56:47,847 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=21, state=SUCCESS; OpenRegionProcedure 9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 in 184 msec 2023-03-20 10:56:47,848 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-03-20 10:56:47,848 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=9fc4d04a6b2183ce1217c4847d39aded, REOPEN/MOVE in 505 msec 2023-03-20 10:56:47,851 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-03-20 10:56:47,851 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 521 msec 2023-03-20 10:56:47,853 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 955 msec 2023-03-20 10:56:47,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-03-20 10:56:47,861 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(139): Connect 0x3b75e45b to 127.0.0.1:60226 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 10:56:47,866 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2d781def, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 10:56:47,867 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(139): Connect 0x6017755a to 127.0.0.1:60226 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 10:56:47,870 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@595355a0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 10:56:47,870 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(139): Connect 0x633c81ba to 127.0.0.1:60226 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 10:56:47,874 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@38da9522, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 10:56:47,874 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(139): Connect 0x661b21c2 to 127.0.0.1:60226 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 10:56:47,879 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@a84dba6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 10:56:47,879 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(139): Connect 0x43a14bcf to 127.0.0.1:60226 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 10:56:47,884 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7c0b7748, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 10:56:47,884 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(139): Connect 0x2c411e08 to 127.0.0.1:60226 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 10:56:47,887 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@bd79dff, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 10:56:47,888 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(139): Connect 0x7605c204 to 127.0.0.1:60226 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 10:56:47,891 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@52902392, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 10:56:47,891 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(139): Connect 0x16505096 to 127.0.0.1:60226 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 10:56:47,895 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4e1da5ba, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 10:56:47,895 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(139): Connect 0x03e1990e to 127.0.0.1:60226 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 10:56:47,899 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@17a1bd04, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 10:56:47,904 DEBUG [hconnection-0x6743a1f9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 10:56:47,905 DEBUG [hconnection-0x7175d325-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 10:56:47,907 DEBUG [hconnection-0x71e7b544-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 10:56:47,908 DEBUG [hconnection-0x71add79-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 10:56:47,908 DEBUG [hconnection-0x6066108c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 10:56:47,908 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:42928, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 10:56:47,910 DEBUG [hconnection-0x4b8d50eb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 10:56:47,911 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:42958, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 10:56:47,913 DEBUG [hconnection-0x56f5dfc-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 10:56:47,915 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:42936, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 10:56:47,915 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:42948, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 10:56:47,915 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:42972, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 10:56:47,915 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:42986, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 10:56:47,915 DEBUG [hconnection-0x6899296c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 10:56:47,917 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43002, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 10:56:47,916 DEBUG [hconnection-0x574aa420-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 10:56:47,920 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43004, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 10:56:47,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:47,930 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43018, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 10:56:47,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:47,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:47,936 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:47,936 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:47,936 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:47,936 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:47,937 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:47,937 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:47,937 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:47,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:47,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:47,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:47,939 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:47,939 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:47,939 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:47,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:47,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:47,939 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:47,939 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:47,943 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:47,943 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:47,943 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:47,943 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:47,943 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:47,943 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9fc4d04a6b2183ce1217c4847d39aded 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-20 10:56:47,944 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9fc4d04a6b2183ce1217c4847d39aded, store=A 2023-03-20 10:56:47,944 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:47,944 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9fc4d04a6b2183ce1217c4847d39aded, store=B 2023-03-20 10:56:47,944 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:47,944 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9fc4d04a6b2183ce1217c4847d39aded, store=C 2023-03-20 10:56:47,944 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:47,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 9fc4d04a6b2183ce1217c4847d39aded 2023-03-20 10:56:47,946 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:47,946 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:47,946 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:47,946 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. started... 2023-03-20 10:56:47,946 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. as already flushing 2023-03-20 10:56:47,946 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:47,947 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:47,947 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:47,947 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:47,948 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:47,948 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:47,952 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:47,952 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:47,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:47,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:47,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:47,953 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:47,953 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:47,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:47,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:47,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:47,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:47,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:47,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:47,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c6029e1[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:47,955 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:47,956 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:47,960 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:47,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:47,960 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:47,960 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:47,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:47,961 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:47,961 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:47,961 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:47,962 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:47,962 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:47,962 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:47,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:47,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:47,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:47,962 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:47,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:47,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:47,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:47,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:47,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:47,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:47,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:47,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:47,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:47,965 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:47,965 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:47,965 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:47,965 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:47,965 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:47,967 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:47,967 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:47,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:47,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:47,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:47,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:47,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:47,967 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:47,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:47,967 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:47,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:47,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:47,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:47,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:47,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:47,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:47,969 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:47,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:47,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:47,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:47,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:47,971 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:47,971 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:47,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:47,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:47,971 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:47,971 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:47,971 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:47,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:47,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:47,971 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:47,971 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:47,971 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:47,971 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:47,971 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:47,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:47,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:47,972 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:56:47,973 DEBUG [Thread-875] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:47,973 DEBUG [Thread-875] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:47,981 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679309807957, max timestamp=1679309807979Num uniques 859; 2023-03-20 10:56:47,981 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679309807957, max timestamp=1679309807979Num uniques 859; 2023-03-20 10:56:47,981 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-20 10:56:47,981 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679309807957, max timestamp=1679309807979Num uniques 859; 2023-03-20 10:56:47,982 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-20 10:56:47,981 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-20 10:56:48,004 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:48,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42936 deadline: 1679309868002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,011 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:48,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42958 deadline: 1679309868004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,011 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:48,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42986 deadline: 1679309868004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,011 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:48,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42972 deadline: 1679309868004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,023 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:48,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43002 deadline: 1679309868020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,073 DEBUG [Thread-875] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:48,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:48,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:48,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:48,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:48,079 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:48,079 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:48,080 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:48,080 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:48,080 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:48,080 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:48,081 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,081 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:48,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:48,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:48,081 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,081 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:48,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:48,082 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:48,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:48,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:48,082 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:48,086 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:48,086 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:48,087 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:48,087 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:48,087 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:48,096 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:48,096 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:48,108 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:48,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42936 deadline: 1679309868106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,108 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:48,109 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. started... 2023-03-20 10:56:48,109 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. as already flushing 2023-03-20 10:56:48,109 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:48,109 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:48,109 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:48,109 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,113 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:48,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42972 deadline: 1679309868113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,114 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:48,114 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:48,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42958 deadline: 1679309868113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42986 deadline: 1679309868113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,121 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:48,121 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:48,122 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:48,122 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:48,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:48,122 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:48,123 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:48,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:48,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:48,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:48,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:48,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:48,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d9576fd[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:48,125 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:48,125 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:48,126 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:48,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:48,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:48,126 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:48,126 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:48,126 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:48,126 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,126 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:48,127 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:48,127 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:48,127 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:48,127 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:48,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:48,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:48,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:48,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:48,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:48,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:48,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,131 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:48,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43002 deadline: 1679309868128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:48,131 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:48,131 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:48,132 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:48,132 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:48,132 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:48,132 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:48,135 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:48,135 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:48,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:48,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:48,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:48,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:48,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:48,135 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:48,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:48,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:48,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:48,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:48,137 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:48,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:48,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:48,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,150 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:48,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:48,154 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,154 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:48,154 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:48,154 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:48,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:48,154 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:56:48,154 DEBUG [Thread-875] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:48,154 DEBUG [Thread-875] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:48,154 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:48,154 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:48,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:48,154 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:48,154 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:48,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:48,154 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:48,154 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:48,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:48,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:48,254 DEBUG [Thread-875] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:48,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:48,259 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:48,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:48,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:48,297 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:48,298 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:48,298 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:48,298 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:48,298 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:48,298 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:48,299 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:48,299 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:48,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:48,299 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,299 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:48,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:48,300 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:48,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:48,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:48,300 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:48,306 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:48,306 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:48,306 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:48,306 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:48,306 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:48,311 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:48,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42936 deadline: 1679309868311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,316 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:48,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42972 deadline: 1679309868315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,317 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:48,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42986 deadline: 1679309868315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,317 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:48,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42958 deadline: 1679309868316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,322 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:48,322 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:48,322 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:48,322 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. started... 2023-03-20 10:56:48,322 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. as already flushing 2023-03-20 10:56:48,322 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:48,322 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:48,322 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:48,322 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,322 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:48,322 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:48,324 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:48,325 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:48,325 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:48,325 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:48,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:48,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:48,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:48,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:48,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:48,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:48,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66aa335f[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:48,327 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:48,327 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:48,328 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:48,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:48,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:48,328 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:48,328 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:48,328 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:48,328 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,328 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:48,330 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,330 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:48,330 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:48,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:48,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:48,330 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:48,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:48,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:48,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:48,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:48,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:48,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:48,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:48,333 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:48,333 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:48,333 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:48,333 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:48,333 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:48,335 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:48,335 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:48,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:48,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:48,335 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:48,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:48,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:48,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:48,335 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:48,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:48,336 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:48,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:48,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43002 deadline: 1679309868334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:48,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:48,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:48,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:48,338 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:48,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:48,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,341 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:48,341 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,341 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:48,341 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:56:48,341 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:48,341 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:48,341 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:48,341 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,341 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:48,341 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:48,341 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:48,341 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:48,341 DEBUG [Thread-875] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:48,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:48,341 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:48,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:48,342 DEBUG [Thread-875] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:48,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:48,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:48,400 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230320bb19f60cf2994be78cda41047bfe3a1a to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320bb19f60cf2994be78cda41047bfe3a1a 2023-03-20 10:56:48,442 DEBUG [Thread-875] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:48,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:48,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:48,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:48,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:48,460 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:48,460 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:48,460 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:48,460 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:48,460 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:48,460 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:48,462 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:48,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:48,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:48,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:48,463 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:48,463 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:48,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:48,464 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:48,464 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,464 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:48,471 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:48,471 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:48,471 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:48,471 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:48,471 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:48,490 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:48,490 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:48,490 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:48,490 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. started... 2023-03-20 10:56:48,490 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. as already flushing 2023-03-20 10:56:48,490 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:48,490 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:48,491 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:48,491 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,506 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:48,506 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:48,508 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,508 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:48,508 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:48,508 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:48,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:48,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:48,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:48,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:48,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:48,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:48,512 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:48,512 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f17035f[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:48,512 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:48,512 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:48,513 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:48,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:48,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:48,514 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:48,514 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:48,514 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:48,514 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,514 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:48,515 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:48,515 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:48,515 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:48,515 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:48,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:48,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:48,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:48,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:48,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:48,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:48,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:48,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:48,528 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:48,528 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:48,528 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:48,528 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:48,528 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:48,535 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:48,535 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:48,535 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:48,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:48,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:48,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:48,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:48,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:48,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:48,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:48,536 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:48,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:48,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:48,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:48,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:48,540 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:48,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:48,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,543 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,544 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:48,544 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:48,544 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,544 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:48,544 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:48,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:48,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:48,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:48,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:48,544 DEBUG [Thread-875] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:48,544 DEBUG [Thread-875] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:48,544 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:56:48,544 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:48,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:48,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:48,544 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:48,545 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:48,545 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:48,616 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:48,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42936 deadline: 1679309868616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,624 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:48,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42958 deadline: 1679309868623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,624 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:48,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42972 deadline: 1679309868623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,644 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:48,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42986 deadline: 1679309868642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,644 DEBUG [Thread-875] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:48,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:48,648 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:48,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43002 deadline: 1679309868648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:48,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:48,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:48,695 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:48,695 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:48,696 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:48,696 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:48,696 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:48,696 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:48,697 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:48,697 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:48,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:48,697 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,698 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:48,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:48,698 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:48,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:48,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:48,698 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:48,718 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:48,718 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:48,718 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:48,718 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:48,718 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:48,746 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:48,746 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:48,753 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:48,753 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. started... 2023-03-20 10:56:48,753 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. as already flushing 2023-03-20 10:56:48,753 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:48,753 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:48,753 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:48,753 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,757 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:48,757 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:48,758 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,759 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:48,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:48,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:48,759 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:48,759 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:48,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:48,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:48,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:48,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:48,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:48,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1245007d[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:48,760 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:48,760 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:48,761 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:48,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:48,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:48,761 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:48,761 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:48,761 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:48,761 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,761 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:48,765 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:48,765 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:48,766 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:48,766 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:48,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:48,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:48,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:48,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:48,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:48,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:48,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:48,781 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:48,781 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:48,781 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:48,781 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:48,781 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:48,781 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:48,783 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:48,783 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:48,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:48,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:48,783 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:48,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:48,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:48,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:48,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:48,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:48,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:48,784 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:48,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:48,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:48,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:48,785 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:48,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:48,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,791 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,791 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:48,791 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:48,791 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:48,791 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:48,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:48,791 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:48,792 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:48,792 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:56:48,792 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:48,792 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:48,791 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:48,792 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:48,792 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:48,792 DEBUG [Thread-875] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:48,792 DEBUG [Thread-875] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:48,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:48,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:48,860 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/A/72f846aad3344212adb40a49efd0b339 2023-03-20 10:56:48,892 DEBUG [Thread-875] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:48,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:48,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:48,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:48,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:48,961 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:48,961 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:48,961 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:48,961 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:48,962 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:48,962 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:48,967 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,967 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:48,968 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:48,967 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:48,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:48,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:48,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:48,969 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:48,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:48,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:48,969 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:48,984 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:48,984 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:48,985 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:48,985 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:48,985 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:48,997 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:48,997 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:48,998 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:48,998 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. started... 2023-03-20 10:56:48,999 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. as already flushing 2023-03-20 10:56:48,999 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:48,999 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:48,999 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:48,999 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:49,006 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:49,006 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:49,008 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:49,008 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:49,008 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:49,009 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:49,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:49,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:49,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:49,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:49,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:49,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:49,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:49,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:49,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:49,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34c1be56[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:49,026 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:49,026 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:49,027 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:49,028 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:49,028 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:49,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:49,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:49,028 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:49,028 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:49,028 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:49,030 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:49,030 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:49,030 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:49,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:49,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:49,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:49,030 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:49,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:49,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:49,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:49,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:49,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:49,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:49,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:49,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:49,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:49,044 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:49,044 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:49,044 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:49,044 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:49,044 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:49,052 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:49,052 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:49,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:49,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:49,053 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:49,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:49,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:49,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:49,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:49,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:49,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:49,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:49,055 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:49,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:49,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:49,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:49,060 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:49,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:49,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:49,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:49,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:49,079 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:49,079 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:49,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:49,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:49,079 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:49,079 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:49,079 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:49,079 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:49,079 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:49,079 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:49,079 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:49,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:49,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:49,081 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:49,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:49,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:49,082 DEBUG [Thread-875] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:49,082 DEBUG [Thread-875] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:49,124 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:49,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42936 deadline: 1679309869122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:49,131 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:49,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42972 deadline: 1679309869129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:49,132 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:49,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42958 deadline: 1679309869129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:49,149 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:49,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42986 deadline: 1679309869149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:49,153 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:49,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43002 deadline: 1679309869153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:49,182 DEBUG [Thread-875] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:49,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:49,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:49,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:49,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:49,214 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:49,214 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:49,214 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:49,214 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:49,217 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:49,217 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:49,222 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:49,222 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:49,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:49,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:49,223 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:49,223 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:49,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:49,223 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:49,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:49,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:49,224 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:49,234 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:49,234 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:49,234 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:49,234 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:49,234 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:49,251 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:49,251 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:49,251 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:49,251 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. started... 2023-03-20 10:56:49,251 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. as already flushing 2023-03-20 10:56:49,251 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:49,252 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:49,252 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:49,252 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:49,252 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:49,252 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:49,255 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:49,256 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:49,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:49,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:49,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:49,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:49,256 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:49,257 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:49,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:49,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:49,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:49,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:49,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:49,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@346535a[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:49,258 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:49,258 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:49,259 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:49,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:49,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:49,259 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:49,259 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:49,259 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:49,260 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:49,260 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:49,261 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:49,261 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:49,261 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:49,261 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:49,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:49,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:49,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:49,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:49,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:49,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:49,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:49,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:49,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:49,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:49,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:49,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:49,264 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:49,264 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:49,264 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:49,264 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:49,264 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:49,265 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:49,265 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:49,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:49,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:49,265 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:49,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:49,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:49,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:49,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:49,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:49,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:49,267 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:49,268 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:49,270 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:49,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:49,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:49,270 DEBUG [Thread-875] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:49,270 DEBUG [Thread-875] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:49,273 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:49,273 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:49,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:49,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:49,273 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:49,273 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:49,273 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:49,273 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:49,273 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:49,273 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:49,273 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:49,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:49,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:49,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:49,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:49,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:49,276 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:56:49,371 DEBUG [Thread-875] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:49,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:49,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:49,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:49,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:49,387 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:49,387 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:49,387 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:49,387 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:49,392 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:49,392 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:49,395 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/B/4583dacb48e043e79cce494d8b32e043 2023-03-20 10:56:49,399 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:49,399 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:49,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:49,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:49,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:49,400 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:49,400 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:49,400 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:49,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:49,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:49,401 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:49,408 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:49,408 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:49,408 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:49,408 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:49,408 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:49,413 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:49,413 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:49,414 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:49,414 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. started... 2023-03-20 10:56:49,414 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. as already flushing 2023-03-20 10:56:49,414 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:49,414 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:49,414 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:49,414 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:49,418 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:49,418 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:49,419 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:49,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:49,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:49,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:49,420 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:49,420 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:49,420 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:49,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:49,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:49,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:49,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:49,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:49,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:49,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@549ca596[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:49,422 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:49,422 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:49,423 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:49,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:49,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:49,423 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:49,423 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:49,423 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:49,425 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:49,425 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:49,425 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:49,425 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:49,425 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:49,426 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:49,426 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:49,426 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:49,426 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:49,426 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:49,426 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:49,429 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:49,429 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:49,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:49,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:49,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:49,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:49,429 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:49,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:49,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:49,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:49,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:49,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:49,430 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:49,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:49,432 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:49,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:49,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:49,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:49,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:49,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:49,436 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:49,436 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:49,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:49,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:49,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:49,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:49,437 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:49,437 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:49,437 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:49,437 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:49,437 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:49,437 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:49,438 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:49,438 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:49,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:49,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:49,438 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:56:49,438 DEBUG [Thread-875] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:49,438 DEBUG [Thread-875] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:49,482 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/C/f8aa35048315471cb9227656cbc0b79b 2023-03-20 10:56:49,489 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/A/72f846aad3344212adb40a49efd0b339 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/72f846aad3344212adb40a49efd0b339 2023-03-20 10:56:49,498 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/72f846aad3344212adb40a49efd0b339, entries=100, sequenceid=15, filesize=18.4 K 2023-03-20 10:56:49,499 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/B/4583dacb48e043e79cce494d8b32e043 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/4583dacb48e043e79cce494d8b32e043 2023-03-20 10:56:49,512 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/4583dacb48e043e79cce494d8b32e043, entries=100, sequenceid=15, filesize=9.3 K 2023-03-20 10:56:49,513 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/C/f8aa35048315471cb9227656cbc0b79b as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/f8aa35048315471cb9227656cbc0b79b 2023-03-20 10:56:49,524 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/f8aa35048315471cb9227656cbc0b79b, entries=100, sequenceid=15, filesize=9.3 K 2023-03-20 10:56:49,525 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 9fc4d04a6b2183ce1217c4847d39aded in 1582ms, sequenceid=15, compaction requested=false 2023-03-20 10:56:49,525 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9fc4d04a6b2183ce1217c4847d39aded: 2023-03-20 10:56:49,538 DEBUG [Thread-875] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:49,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:49,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:49,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:49,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:49,546 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:49,546 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:49,547 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:49,547 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:49,547 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:49,547 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:49,548 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:49,548 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:49,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:49,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:49,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:49,548 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:49,548 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:49,549 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:49,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:49,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:49,549 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:49,571 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:49,571 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:49,571 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:49,571 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:49,571 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:49,576 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:49,577 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:49,577 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:49,577 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. started... 2023-03-20 10:56:49,577 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9fc4d04a6b2183ce1217c4847d39aded 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-03-20 10:56:49,577 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9fc4d04a6b2183ce1217c4847d39aded, store=A 2023-03-20 10:56:49,577 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:56:49,578 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9fc4d04a6b2183ce1217c4847d39aded, store=B 2023-03-20 10:56:49,578 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:56:49,578 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9fc4d04a6b2183ce1217c4847d39aded, store=C 2023-03-20 10:56:49,578 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:56:49,659 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032003d761d31d64421f9e76ac5710f35330 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032003d761d31d64421f9e76ac5710f35330 2023-03-20 10:56:49,673 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=51, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/A/803bce3658e94a5bbf95a9784858871e 2023-03-20 10:56:49,859 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-20 10:56:50,135 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/B/59de0e0eda424b9ca26438de8fca4f2f 2023-03-20 10:56:50,145 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. as already flushing 2023-03-20 10:56:50,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 9fc4d04a6b2183ce1217c4847d39aded 2023-03-20 10:56:50,164 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:50,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42972 deadline: 1679309870157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:50,165 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:50,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42936 deadline: 1679309870160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:50,168 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:50,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43002 deadline: 1679309870161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:50,170 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:50,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42986 deadline: 1679309870163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:50,170 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:50,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42958 deadline: 1679309870165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:50,272 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:50,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42936 deadline: 1679309870267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:50,273 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:50,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42972 deadline: 1679309870267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:50,275 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:50,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42986 deadline: 1679309870271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:50,286 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:50,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42958 deadline: 1679309870282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:50,477 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:50,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42936 deadline: 1679309870476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:50,485 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:50,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42972 deadline: 1679309870479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:50,492 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:50,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42986 deadline: 1679309870478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:50,493 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:50,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42958 deadline: 1679309870490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:50,617 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/C/368d499ef675406bbc45823c3716be61 2023-03-20 10:56:50,624 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/A/803bce3658e94a5bbf95a9784858871e as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/803bce3658e94a5bbf95a9784858871e 2023-03-20 10:56:50,630 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/803bce3658e94a5bbf95a9784858871e, entries=150, sequenceid=51, filesize=25.1 K 2023-03-20 10:56:50,633 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/B/59de0e0eda424b9ca26438de8fca4f2f as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/59de0e0eda424b9ca26438de8fca4f2f 2023-03-20 10:56:50,653 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/59de0e0eda424b9ca26438de8fca4f2f, entries=150, sequenceid=51, filesize=11.6 K 2023-03-20 10:56:50,655 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/C/368d499ef675406bbc45823c3716be61 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/368d499ef675406bbc45823c3716be61 2023-03-20 10:56:50,669 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/368d499ef675406bbc45823c3716be61, entries=150, sequenceid=51, filesize=11.6 K 2023-03-20 10:56:50,670 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 9fc4d04a6b2183ce1217c4847d39aded in 1093ms, sequenceid=51, compaction requested=false 2023-03-20 10:56:50,670 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9fc4d04a6b2183ce1217c4847d39aded: 2023-03-20 10:56:50,670 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:50,670 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:50,670 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:50,670 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:50,695 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:50,695 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:50,711 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:50,714 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:50,714 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:50,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:50,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:50,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:50,714 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:50,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:50,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:50,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:50,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:50,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:50,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:50,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ee01190[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:50,729 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:50,729 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:50,730 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:50,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:50,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:50,730 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:50,730 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:50,730 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:50,731 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:50,731 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:50,731 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:50,732 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:50,733 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:50,734 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:50,734 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:50,734 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:50,734 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:50,734 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:50,734 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:50,737 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:50,737 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:50,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:50,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:50,737 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:50,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:50,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:50,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:50,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:50,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:50,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:50,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:50,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:50,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:50,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:50,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:50,740 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:50,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:50,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:50,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:50,742 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:50,753 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:50,753 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:50,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:50,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:50,753 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:50,753 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:50,753 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:50,753 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:50,753 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:50,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:50,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:50,753 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:50,753 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:50,753 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:50,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:50,753 DEBUG [Thread-875] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:50,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:50,754 DEBUG [Thread-875] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:50,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 9fc4d04a6b2183ce1217c4847d39aded 2023-03-20 10:56:50,784 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9fc4d04a6b2183ce1217c4847d39aded 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-20 10:56:50,785 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9fc4d04a6b2183ce1217c4847d39aded, store=A 2023-03-20 10:56:50,785 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:50,785 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9fc4d04a6b2183ce1217c4847d39aded, store=B 2023-03-20 10:56:50,785 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:50,785 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9fc4d04a6b2183ce1217c4847d39aded, store=C 2023-03-20 10:56:50,785 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:50,850 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:50,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42972 deadline: 1679309870844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:50,851 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679309810810, max timestamp=1679309810848Num uniques 859; 2023-03-20 10:56:50,851 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679309810810, max timestamp=1679309810848Num uniques 859; 2023-03-20 10:56:50,851 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-20 10:56:50,851 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679309810810, max timestamp=1679309810848Num uniques 859; 2023-03-20 10:56:50,852 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-20 10:56:50,851 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-20 10:56:50,854 DEBUG [Thread-875] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:50,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:50,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:50,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:50,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:50,861 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:50,861 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:50,861 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:50,861 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:50,862 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:50,862 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:50,862 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:50,862 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:50,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:50,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:50,863 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:50,863 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:50,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:50,864 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:50,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:50,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:50,864 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:50,866 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:50,866 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:50,866 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:50,866 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:50,866 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:50,867 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:50,867 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:50,868 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:50,868 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. started... 2023-03-20 10:56:50,868 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. as already flushing 2023-03-20 10:56:50,868 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:50,868 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:50,868 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:50,868 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:50,868 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:50,868 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:50,903 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:50,903 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:50,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:50,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:50,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:50,903 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:50,903 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:50,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:50,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:50,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:50,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:50,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:50,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:50,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@576a3114[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:50,906 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:50,906 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:50,907 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:50,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:50,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:50,907 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:50,907 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:50,907 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:50,907 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:50,907 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:50,908 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:50,908 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:50,908 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:50,909 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:50,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:50,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:50,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:50,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:50,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:50,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:50,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:50,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:50,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:50,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:50,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:50,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:50,916 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:50,916 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:50,916 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:50,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42986 deadline: 1679309870911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:50,921 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:50,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42936 deadline: 1679309870913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:50,916 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:50,922 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:50,922 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:50,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42958 deadline: 1679309870913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:50,922 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:50,928 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:50,928 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:50,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:50,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:50,929 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:50,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:50,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:50,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:50,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:50,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:50,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:50,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:50,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:50,930 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:50,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:50,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:50,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:50,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:50,931 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:50,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:50,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:50,933 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:50,933 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:50,933 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:50,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:50,933 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:50,933 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:50,933 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:50,933 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:50,933 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:50,933 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:50,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:50,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:50,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:50,933 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:50,933 DEBUG [Thread-875] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:50,933 DEBUG [Thread-875] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:50,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:50,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:50,954 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:50,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42972 deadline: 1679309870952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,024 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:51,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42986 deadline: 1679309871019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,029 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:51,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42958 deadline: 1679309871028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,029 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:51,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42936 deadline: 1679309871027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,035 DEBUG [Thread-875] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:51,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:51,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:51,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:51,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:51,063 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:51,064 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:51,064 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:51,064 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:51,065 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:51,065 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:51,066 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:51,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:51,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:51,067 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:51,068 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:51,068 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,068 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:51,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:51,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:51,069 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:51,071 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:51,072 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:51,072 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:51,072 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:51,072 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:51,073 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:51,073 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:51,074 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:51,074 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. started... 2023-03-20 10:56:51,074 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. as already flushing 2023-03-20 10:56:51,074 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:51,074 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:51,074 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:51,074 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,074 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:51,074 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:51,077 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:51,077 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:51,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:51,077 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:51,077 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:51,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:51,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:51,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:51,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:51,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:51,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66876420[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:51,079 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:51,079 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:51,085 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:51,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:51,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:51,085 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:51,086 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:51,086 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:51,086 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,086 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:51,087 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,087 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:51,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:51,087 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:51,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:51,087 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:51,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:51,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:51,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:51,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:51,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:51,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:51,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:51,090 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:51,090 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:51,090 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:51,090 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:51,090 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:51,102 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:51,102 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:51,102 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:51,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:51,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:51,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:51,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:51,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:51,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:51,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:51,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:51,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:51,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:51,104 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:51,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:51,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:51,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:51,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,110 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,118 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:51,118 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:51,119 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:51,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:51,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:51,119 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,119 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:51,119 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:51,119 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,119 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:51,119 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:51,119 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:51,120 DEBUG [Thread-875] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:51,120 DEBUG [Thread-875] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:51,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:51,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:51,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:51,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:51,159 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:51,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42972 deadline: 1679309871159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,220 DEBUG [Thread-875] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:51,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:51,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:51,229 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:51,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42986 deadline: 1679309871229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,233 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:51,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42958 deadline: 1679309871232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,234 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:51,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42936 deadline: 1679309871233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:51,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:51,238 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:51,238 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:51,238 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:51,238 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:51,239 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:51,239 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:51,239 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230320af01a3bab4ee482283ba4cc17865bfa8 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320af01a3bab4ee482283ba4cc17865bfa8 2023-03-20 10:56:51,240 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,240 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:51,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:51,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:51,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:51,240 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,240 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:51,242 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:51,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:51,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:51,243 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:51,245 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:51,245 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:51,245 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:51,245 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:51,245 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:51,246 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:51,246 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:51,246 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:51,246 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. started... 2023-03-20 10:56:51,246 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. as already flushing 2023-03-20 10:56:51,246 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:51,246 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:51,246 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:51,246 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,246 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:51,246 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:51,252 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:51,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:51,252 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:51,254 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:51,254 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:51,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:51,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:51,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:51,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:51,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:51,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5706cd73[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:51,256 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:51,256 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:51,261 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:51,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:51,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:51,262 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:51,262 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:51,262 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:51,262 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,262 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:51,268 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:51,268 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,268 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:51,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:51,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:51,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:51,269 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:51,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:51,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:51,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:51,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:51,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:51,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:51,270 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:51,270 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:51,271 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:51,271 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:51,271 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:51,277 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:51,277 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:51,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:51,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:51,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:51,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:51,277 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:51,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:51,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:51,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:51,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:51,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:51,278 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:51,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:51,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:51,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,279 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:51,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:51,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,286 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:51,286 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,286 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:51,286 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:51,286 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:51,286 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:51,286 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:51,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:51,286 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:51,286 DEBUG [Thread-875] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:51,286 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:51,287 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:51,287 DEBUG [Thread-875] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:51,287 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:51,287 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:51,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:51,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:51,387 DEBUG [Thread-875] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:51,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:51,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:51,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:51,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:51,392 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:51,392 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:51,392 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:51,392 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:51,393 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:51,393 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:51,394 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:51,394 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:51,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:51,394 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,394 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:51,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:51,395 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:51,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:51,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:51,395 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:51,398 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:51,398 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:51,398 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:51,398 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:51,398 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:51,399 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:51,399 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:51,399 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:51,399 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. started... 2023-03-20 10:56:51,399 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. as already flushing 2023-03-20 10:56:51,399 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:51,399 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:51,399 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:51,399 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,399 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:51,399 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:51,400 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,400 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:51,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:51,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:51,401 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:51,401 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:51,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:51,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:51,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:51,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:51,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:51,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39843a92[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:51,402 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:51,402 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:51,403 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:51,403 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,403 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:51,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:51,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:51,404 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:51,404 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:51,404 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:51,412 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:51,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:51,412 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:51,412 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:51,412 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:51,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:51,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:51,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:51,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:51,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:51,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:51,416 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:51,416 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:51,416 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:51,416 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:51,416 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:51,416 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:51,418 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:51,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:51,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:51,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:51,418 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,418 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:51,419 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:51,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:51,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:51,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:51,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:51,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:51,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:51,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:51,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:51,420 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:51,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:51,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:51,422 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:51,422 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:51,422 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,422 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:51,422 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:51,422 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,422 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:51,422 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:51,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:51,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:51,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:51,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:51,422 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:51,423 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:51,423 DEBUG [Thread-875] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:51,423 DEBUG [Thread-875] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:51,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:51,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:51,423 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:56:51,465 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:51,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42972 deadline: 1679309871463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,523 DEBUG [Thread-875] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:51,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:51,534 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:51,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42986 deadline: 1679309871533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,539 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:51,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42958 deadline: 1679309871539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,545 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:51,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42936 deadline: 1679309871542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:51,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:51,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:51,558 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:51,558 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:51,558 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:51,558 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:51,562 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:51,562 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:51,564 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:51,564 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:51,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:51,564 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,565 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:51,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:51,565 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:51,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:51,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:51,565 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:51,566 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:51,566 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:51,566 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:51,566 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:51,566 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:51,566 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:51,566 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:51,566 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:51,566 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. started... 2023-03-20 10:56:51,566 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. as already flushing 2023-03-20 10:56:51,566 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:51,566 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:51,566 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:51,567 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,567 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:51,567 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:51,571 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,571 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:51,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:51,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:51,571 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:51,571 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:51,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:51,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:51,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:51,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:51,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:51,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f605e27[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:51,573 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:51,573 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:51,574 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:51,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:51,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:51,574 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:51,574 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,574 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:51,574 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:51,574 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:51,575 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:51,575 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,575 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:51,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:51,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:51,575 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:51,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:51,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:51,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:51,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:51,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:51,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:51,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:51,577 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:51,577 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:51,577 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:51,577 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:51,577 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:51,580 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:51,580 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:51,580 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:51,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:51,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:51,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:51,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:51,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:51,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:51,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:51,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:51,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:51,589 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:51,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:51,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:51,593 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:51,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:51,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,610 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:51,610 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:51,610 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,610 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:51,610 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:51,610 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,610 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:51,610 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:51,610 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:51,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:51,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:51,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:51,610 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:51,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:51,611 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:56:51,611 DEBUG [Thread-875] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:51,611 DEBUG [Thread-875] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:51,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:51,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:51,665 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=62, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/A/8be7d7523828421c88108f18d94b744b 2023-03-20 10:56:51,718 DEBUG [Thread-875] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:51,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:51,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:51,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:51,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:51,735 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:51,735 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:51,735 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:51,735 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:51,735 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:51,735 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:51,736 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:51,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:51,737 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:51,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:51,738 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:51,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:51,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:51,738 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:51,739 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:51,739 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:51,739 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:51,739 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:51,739 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:51,739 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:51,739 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,739 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:51,739 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:51,739 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:51,739 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. started... 2023-03-20 10:56:51,739 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. as already flushing 2023-03-20 10:56:51,739 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:51,739 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:51,739 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:51,739 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,740 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:51,740 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:51,741 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:51,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:51,741 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:51,741 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:51,741 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:51,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:51,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:51,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:51,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:51,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:51,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@629b8b8f[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:51,742 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:51,742 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:51,743 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:51,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:51,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:51,744 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:51,744 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:51,744 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:51,746 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:51,746 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:51,746 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:51,748 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,750 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:51,750 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:51,750 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:51,750 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:51,750 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:51,750 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:51,750 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:51,751 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:51,751 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:51,751 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:51,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:51,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:51,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:51,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:51,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:51,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:51,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:51,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:51,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:51,752 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:51,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:51,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:51,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,754 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:51,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:51,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,757 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,757 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:51,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:51,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:51,757 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:51,757 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:51,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:51,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:51,757 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:51,757 DEBUG [Thread-875] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:51,757 DEBUG [Thread-875] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:51,757 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,757 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:51,757 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:51,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:51,757 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:51,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:51,758 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:51,857 DEBUG [Thread-875] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:51,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:51,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:51,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:51,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:51,870 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:51,870 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:51,870 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:51,870 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:51,871 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:51,871 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:51,872 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,872 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:51,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:51,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:51,872 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,873 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:51,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:51,873 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:51,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:51,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:51,874 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:51,878 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:51,878 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:51,878 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:51,878 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:51,878 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:51,882 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:51,882 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:51,882 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:51,882 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. started... 2023-03-20 10:56:51,882 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. as already flushing 2023-03-20 10:56:51,882 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:51,882 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:51,882 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:51,882 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,882 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:51,882 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:51,893 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,894 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:51,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:51,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:51,894 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:51,894 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:51,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:51,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:51,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:51,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:51,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:51,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@397affa[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:51,899 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:51,899 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:51,904 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,904 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:51,906 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:51,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:51,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:51,906 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:51,906 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:51,906 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:51,911 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:51,911 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:51,912 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,912 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:51,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:51,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:51,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:51,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:51,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:51,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:51,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:51,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:51,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:51,914 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:51,914 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:51,914 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:51,914 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:51,915 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:51,919 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:51,919 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:51,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:51,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:51,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:51,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:51,919 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:51,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:51,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:51,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:51,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:51,919 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:51,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:51,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:51,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:51,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,920 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:51,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:51,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,932 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:51,932 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:51,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:51,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:51,932 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,932 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:51,932 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:51,932 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:51,932 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:51,932 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:51,932 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:51,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:51,933 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:51,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:51,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:51,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:51,933 DEBUG [Thread-875] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:51,933 DEBUG [Thread-875] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:51,969 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:51,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42972 deadline: 1679309871966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:52,033 DEBUG [Thread-875] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:52,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:52,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:52,040 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:52,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42986 deadline: 1679309872038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:52,046 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:52,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:52,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42958 deadline: 1679309872044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:52,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:52,046 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:52,046 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:52,046 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:52,046 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:52,048 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:52,048 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:52,048 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:52,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:52,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:52,049 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:52,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:52,049 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:52,049 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:52,049 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:52,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:52,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:52,050 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:52,050 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:52,050 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:52,050 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:52,051 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:52,051 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:52,053 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:52,053 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:52,053 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:52,053 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. started... 2023-03-20 10:56:52,053 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. as already flushing 2023-03-20 10:56:52,053 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:52,053 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:52,053 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:52,053 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:52,053 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:52,053 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:52,057 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:52,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42936 deadline: 1679309872051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:52,060 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:52,061 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:52,061 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:52,062 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:52,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:52,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:52,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:52,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:52,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:52,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:52,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:52,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:52,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:52,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b5f45b0[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:52,066 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:52,066 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:52,073 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:52,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:52,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:52,074 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:52,074 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:52,078 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:52,074 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:52,079 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:52,080 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:52,081 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:52,081 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:52,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:52,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:52,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:52,081 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:52,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:52,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:52,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:52,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:52,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:52,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:52,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:52,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:52,084 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:52,084 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:52,084 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:52,084 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:52,084 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:52,084 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:52,085 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:52,085 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:52,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:52,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:52,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:52,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:52,086 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:52,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:52,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:52,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:52,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:52,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:52,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:52,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:52,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:52,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:52,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:52,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:52,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:52,092 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:52,096 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:52,099 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:52,099 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:52,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:52,099 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:52,099 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:52,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:52,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:52,099 DEBUG [Thread-875] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:52,099 DEBUG [Thread-875] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:52,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:52,099 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:52,100 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:52,100 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:52,100 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:52,100 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:52,100 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:52,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:52,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:52,133 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=62 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/B/8bfc066fd2c24d14a73c8c243b36c895 2023-03-20 10:56:52,176 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:52,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43002 deadline: 1679309872174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:52,179 DEBUG [Thread-871] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4159 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded., hostname=jenkins-hbase20.apache.org,36041,1679309777062, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-20 10:56:52,199 DEBUG [Thread-875] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:52,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:52,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:52,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:52,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:52,214 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:52,214 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:52,214 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:52,214 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:52,217 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:52,217 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:52,218 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:52,218 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:52,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:52,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:52,218 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:52,218 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:52,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:52,219 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:52,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:52,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:52,220 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:52,222 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:52,222 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:52,222 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:52,222 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:52,222 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:52,227 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:52,227 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:52,227 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:52,227 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. started... 2023-03-20 10:56:52,227 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. as already flushing 2023-03-20 10:56:52,227 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:52,227 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:52,227 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:52,227 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:52,227 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:52,227 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:52,228 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:52,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:52,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:52,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:52,229 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:52,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:52,229 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:52,229 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:52,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:52,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:52,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:52,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:52,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:52,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2673b19a[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:52,231 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:52,231 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:52,235 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:52,235 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:52,238 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:52,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:52,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:52,238 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:52,238 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:52,238 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:52,239 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:52,240 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:52,240 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:52,242 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:52,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:52,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:52,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:52,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:52,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:52,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:52,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:52,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:52,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:52,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:52,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:52,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:52,262 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:52,262 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:52,262 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:52,262 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:52,262 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:52,264 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:52,264 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:52,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:52,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:52,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:52,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:52,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:52,264 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:52,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:52,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:52,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:52,264 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:52,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:52,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:52,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:52,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:52,266 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:52,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:52,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:52,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:52,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:52,270 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:52,270 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:52,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:52,270 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:52,270 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:52,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:52,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:52,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:52,270 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:52,270 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:52,271 DEBUG [Thread-875] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:52,271 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:52,271 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:52,271 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:52,271 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:52,271 DEBUG [Thread-875] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:52,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:52,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:52,371 DEBUG [Thread-875] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:52,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:52,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:52,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:52,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:52,374 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:52,374 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:52,374 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:52,374 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:52,375 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:52,375 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:52,524 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:52,524 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:52,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:52,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:52,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:52,525 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:52,525 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:52,525 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:52,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:52,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:52,526 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:52,536 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:52,536 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:52,536 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:52,536 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:52,536 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:52,548 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:52,548 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:52,548 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:52,548 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. started... 2023-03-20 10:56:52,548 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. as already flushing 2023-03-20 10:56:52,548 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:52,548 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:52,548 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:52,548 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:52,548 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:52,548 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:52,550 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:52,550 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:52,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:52,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:52,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:52,550 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:52,550 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:52,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:52,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:52,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:52,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:52,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:52,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:52,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@188a9128[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:52,553 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:52,553 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:52,554 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=62 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/C/700517f9411f4e8897bceb64a490f1f6 2023-03-20 10:56:52,554 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:52,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:52,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:52,554 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:52,554 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:52,554 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:52,554 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:52,554 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:52,555 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:52,555 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:52,555 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:52,555 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:52,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:52,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:52,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:52,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:52,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:52,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:52,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:52,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:52,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:52,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:52,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:52,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:52,560 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:52,560 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:52,560 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:52,560 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:52,560 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:52,562 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:52,562 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:52,562 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:52,562 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/A/8be7d7523828421c88108f18d94b744b as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/8be7d7523828421c88108f18d94b744b 2023-03-20 10:56:52,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:52,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:52,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:52,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:52,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:52,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:52,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:52,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:52,563 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:52,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:52,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:52,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:52,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:52,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:52,565 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:52,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:52,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:52,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:52,568 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:52,568 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:52,568 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:52,568 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:52,568 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:52,568 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:52,568 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:52,568 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:52,568 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:52,568 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:52,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:52,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:52,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:52,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:52,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:52,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:52,569 DEBUG [Thread-875] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:52,569 DEBUG [Thread-875] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:52,584 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/8be7d7523828421c88108f18d94b744b, entries=200, sequenceid=62, filesize=31.9 K 2023-03-20 10:56:52,586 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/B/8bfc066fd2c24d14a73c8c243b36c895 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/8bfc066fd2c24d14a73c8c243b36c895 2023-03-20 10:56:52,595 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/8bfc066fd2c24d14a73c8c243b36c895, entries=150, sequenceid=62, filesize=11.6 K 2023-03-20 10:56:52,596 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/C/700517f9411f4e8897bceb64a490f1f6 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/700517f9411f4e8897bceb64a490f1f6 2023-03-20 10:56:52,636 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/700517f9411f4e8897bceb64a490f1f6, entries=150, sequenceid=62, filesize=11.6 K 2023-03-20 10:56:52,638 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 9fc4d04a6b2183ce1217c4847d39aded in 1854ms, sequenceid=62, compaction requested=true 2023-03-20 10:56:52,638 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9fc4d04a6b2183ce1217c4847d39aded: 2023-03-20 10:56:52,638 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:56:52,638 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-20 10:56:52,639 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-03-20 10:56:52,639 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 10:56:52,639 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 10:56:52,641 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 77230 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 10:56:52,641 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1912): 9fc4d04a6b2183ce1217c4847d39aded/A is initiating minor compaction (all files) 2023-03-20 10:56:52,641 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9fc4d04a6b2183ce1217c4847d39aded/A in TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:52,641 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 10:56:52,641 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/72f846aad3344212adb40a49efd0b339, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/803bce3658e94a5bbf95a9784858871e, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/8be7d7523828421c88108f18d94b744b] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp, totalSize=75.4 K 2023-03-20 10:56:52,641 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): 9fc4d04a6b2183ce1217c4847d39aded/C is initiating minor compaction (all files) 2023-03-20 10:56:52,641 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9fc4d04a6b2183ce1217c4847d39aded/C in TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:52,641 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/f8aa35048315471cb9227656cbc0b79b, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/368d499ef675406bbc45823c3716be61, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/700517f9411f4e8897bceb64a490f1f6] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp, totalSize=32.5 K 2023-03-20 10:56:52,642 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 72f846aad3344212adb40a49efd0b339, keycount=100, bloomtype=ROW, size=18.4 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1679309807939 2023-03-20 10:56:52,642 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting f8aa35048315471cb9227656cbc0b79b, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1679309807939 2023-03-20 10:56:52,642 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 803bce3658e94a5bbf95a9784858871e, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1679309807995 2023-03-20 10:56:52,643 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 368d499ef675406bbc45823c3716be61, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1679309807995 2023-03-20 10:56:52,644 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 700517f9411f4e8897bceb64a490f1f6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1679309810155 2023-03-20 10:56:52,646 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 8be7d7523828421c88108f18d94b744b, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1679309810155 2023-03-20 10:56:52,669 DEBUG [Thread-875] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:52,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:52,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:52,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:52,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:52,682 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:52,682 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:52,682 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:52,682 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:52,683 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:52,683 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:52,684 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:52,684 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:52,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:52,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:52,684 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:52,684 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:52,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:52,685 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:52,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:52,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:52,685 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:52,699 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): 9fc4d04a6b2183ce1217c4847d39aded#C#compaction#120 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:56:52,713 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:52,713 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:52,713 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:52,713 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:52,713 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:52,726 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:52,726 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:52,726 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:52,726 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. started... 2023-03-20 10:56:52,726 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9fc4d04a6b2183ce1217c4847d39aded 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-03-20 10:56:52,727 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9fc4d04a6b2183ce1217c4847d39aded, store=A 2023-03-20 10:56:52,727 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:56:52,727 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9fc4d04a6b2183ce1217c4847d39aded, store=B 2023-03-20 10:56:52,727 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:56:52,727 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9fc4d04a6b2183ce1217c4847d39aded, store=C 2023-03-20 10:56:52,727 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:56:52,805 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9fc4d04a6b2183ce1217c4847d39aded#A#compaction#121 average throughput is 0.20 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:56:52,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 9fc4d04a6b2183ce1217c4847d39aded 2023-03-20 10:56:52,982 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. as already flushing 2023-03-20 10:56:53,034 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:53,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42972 deadline: 1679309873026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:53,056 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:53,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42958 deadline: 1679309873051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:53,060 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:53,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42986 deadline: 1679309873057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:53,069 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:53,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42936 deadline: 1679309873068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:53,141 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:53,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42972 deadline: 1679309873136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:53,216 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/C/c719bc090a4a49ed9f7725a2c3f864ab as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/c719bc090a4a49ed9f7725a2c3f864ab 2023-03-20 10:56:53,227 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9fc4d04a6b2183ce1217c4847d39aded/C of 9fc4d04a6b2183ce1217c4847d39aded into c719bc090a4a49ed9f7725a2c3f864ab(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:56:53,227 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9fc4d04a6b2183ce1217c4847d39aded: 2023-03-20 10:56:53,227 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded., storeName=9fc4d04a6b2183ce1217c4847d39aded/C, priority=13, startTime=1679309812639; duration=0sec 2023-03-20 10:56:53,227 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:56:53,228 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 10:56:53,233 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 10:56:53,233 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): 9fc4d04a6b2183ce1217c4847d39aded/B is initiating minor compaction (all files) 2023-03-20 10:56:53,233 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9fc4d04a6b2183ce1217c4847d39aded/B in TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:53,234 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/4583dacb48e043e79cce494d8b32e043, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/59de0e0eda424b9ca26438de8fca4f2f, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/8bfc066fd2c24d14a73c8c243b36c895] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp, totalSize=32.5 K 2023-03-20 10:56:53,235 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 4583dacb48e043e79cce494d8b32e043, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1679309807939 2023-03-20 10:56:53,236 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 59de0e0eda424b9ca26438de8fca4f2f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1679309807995 2023-03-20 10:56:53,237 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 8bfc066fd2c24d14a73c8c243b36c895, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1679309810155 2023-03-20 10:56:53,265 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303206438eb747679466683798447fd618194 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303206438eb747679466683798447fd618194 2023-03-20 10:56:53,267 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): 9fc4d04a6b2183ce1217c4847d39aded#B#compaction#123 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:56:53,346 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:53,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42972 deadline: 1679309873343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:53,375 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/A/9c31b8bb7b964321af7a3300a48b0852 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/9c31b8bb7b964321af7a3300a48b0852 2023-03-20 10:56:53,383 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9fc4d04a6b2183ce1217c4847d39aded/A of 9fc4d04a6b2183ce1217c4847d39aded into 9c31b8bb7b964321af7a3300a48b0852(size=25.2 K), total size for store is 25.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:56:53,383 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9fc4d04a6b2183ce1217c4847d39aded: 2023-03-20 10:56:53,383 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded., storeName=9fc4d04a6b2183ce1217c4847d39aded/A, priority=13, startTime=1679309812638; duration=0sec 2023-03-20 10:56:53,383 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:56:53,654 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:53,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42972 deadline: 1679309873649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:53,712 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=97, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/A/df09ecfeda5f41758eca14656efe0490 2023-03-20 10:56:53,752 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/B/6a68e3e657294e52a8fcc770a5f7df9b as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/6a68e3e657294e52a8fcc770a5f7df9b 2023-03-20 10:56:53,760 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9fc4d04a6b2183ce1217c4847d39aded/B of 9fc4d04a6b2183ce1217c4847d39aded into 6a68e3e657294e52a8fcc770a5f7df9b(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:56:53,761 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9fc4d04a6b2183ce1217c4847d39aded: 2023-03-20 10:56:53,761 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded., storeName=9fc4d04a6b2183ce1217c4847d39aded/B, priority=13, startTime=1679309812638; duration=0sec 2023-03-20 10:56:53,761 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:56:54,162 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:54,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42972 deadline: 1679309874160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:54,167 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=97 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/B/be09314619df44d3832029fe8762e5c6 2023-03-20 10:56:54,219 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=97 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/C/5a3487b49be6401a8ae673f5b5660673 2023-03-20 10:56:54,244 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/A/df09ecfeda5f41758eca14656efe0490 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/df09ecfeda5f41758eca14656efe0490 2023-03-20 10:56:54,250 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/df09ecfeda5f41758eca14656efe0490, entries=150, sequenceid=97, filesize=25.1 K 2023-03-20 10:56:54,254 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/B/be09314619df44d3832029fe8762e5c6 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/be09314619df44d3832029fe8762e5c6 2023-03-20 10:56:54,260 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/be09314619df44d3832029fe8762e5c6, entries=150, sequenceid=97, filesize=11.6 K 2023-03-20 10:56:54,261 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/C/5a3487b49be6401a8ae673f5b5660673 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/5a3487b49be6401a8ae673f5b5660673 2023-03-20 10:56:54,288 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/5a3487b49be6401a8ae673f5b5660673, entries=150, sequenceid=97, filesize=11.6 K 2023-03-20 10:56:54,290 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for 9fc4d04a6b2183ce1217c4847d39aded in 1563ms, sequenceid=97, compaction requested=false 2023-03-20 10:56:54,290 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9fc4d04a6b2183ce1217c4847d39aded: 2023-03-20 10:56:54,290 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:54,290 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:54,290 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:54,290 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:54,290 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:54,290 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:54,291 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:54,292 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:54,292 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:54,294 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:54,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:54,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:54,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:54,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:54,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:54,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:54,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:54,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:54,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:54,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@265c15bc[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:54,295 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:54,296 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:54,296 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:54,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:54,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:54,296 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:54,296 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:54,296 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:54,297 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:54,297 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:54,297 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:54,297 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:54,297 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:54,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:54,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:54,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:54,298 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:54,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:54,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:54,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:54,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:54,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:54,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:54,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:54,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:54,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:54,302 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:54,302 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:54,302 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:54,302 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:54,302 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:54,303 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:54,303 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:54,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:54,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:54,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:54,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:54,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:54,303 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:54,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:54,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:54,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:54,304 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:54,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:54,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:54,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:54,306 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:54,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:54,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:54,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:54,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:54,310 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:54,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:54,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:54,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:54,310 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:54,310 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:54,310 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:54,310 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:54,310 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:54,310 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:54,310 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:54,310 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:54,310 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:54,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:54,310 DEBUG [Thread-875] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:54,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:54,310 DEBUG [Thread-875] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:54,311 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:56:54,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:54,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:54,411 DEBUG [Thread-875] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:54,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:54,412 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:54,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:54,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:54,413 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:54,413 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:54,413 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:54,413 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:54,414 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:54,414 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:54,417 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:54,419 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:54,419 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:54,422 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:54,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:54,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:54,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:54,423 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:54,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:54,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:54,423 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:54,426 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:54,426 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:54,426 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:54,426 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:54,426 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:54,428 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:54,428 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:54,428 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:54,428 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. started... 2023-03-20 10:56:54,428 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9fc4d04a6b2183ce1217c4847d39aded 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-20 10:56:54,428 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9fc4d04a6b2183ce1217c4847d39aded, store=A 2023-03-20 10:56:54,428 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:54,429 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9fc4d04a6b2183ce1217c4847d39aded, store=B 2023-03-20 10:56:54,429 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:54,429 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9fc4d04a6b2183ce1217c4847d39aded, store=C 2023-03-20 10:56:54,429 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:54,855 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303207c78023f78fc40399ad6718b469407a0 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303207c78023f78fc40399ad6718b469407a0 2023-03-20 10:56:55,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 9fc4d04a6b2183ce1217c4847d39aded 2023-03-20 10:56:55,062 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. as already flushing 2023-03-20 10:56:55,111 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:55,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42986 deadline: 1679309875107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,112 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679309815096, max timestamp=1679309815109Num uniques 859; 2023-03-20 10:56:55,112 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-20 10:56:55,113 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679309815096, max timestamp=1679309815109Num uniques 859; 2023-03-20 10:56:55,113 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-20 10:56:55,114 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679309815096, max timestamp=1679309815109Num uniques 859; 2023-03-20 10:56:55,114 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-20 10:56:55,160 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:55,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42936 deadline: 1679309875157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,167 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:55,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42958 deadline: 1679309875166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,170 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:55,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42972 deadline: 1679309875168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,215 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:55,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42986 deadline: 1679309875214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,267 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:55,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42936 deadline: 1679309875266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,268 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:55,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42958 deadline: 1679309875268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,274 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=111, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/A/68564e81a07c46789d63bd6b483206a0 2023-03-20 10:56:55,296 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=111 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/B/4be62bbe615a418296b8f54c7e4ddc16 2023-03-20 10:56:55,341 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=111 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/C/d6be2ed31c414b00a6f5ba349909f7f2 2023-03-20 10:56:55,360 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/A/68564e81a07c46789d63bd6b483206a0 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/68564e81a07c46789d63bd6b483206a0 2023-03-20 10:56:55,390 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/68564e81a07c46789d63bd6b483206a0, entries=150, sequenceid=111, filesize=25.1 K 2023-03-20 10:56:55,394 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/B/4be62bbe615a418296b8f54c7e4ddc16 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/4be62bbe615a418296b8f54c7e4ddc16 2023-03-20 10:56:55,401 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/4be62bbe615a418296b8f54c7e4ddc16, entries=150, sequenceid=111, filesize=11.6 K 2023-03-20 10:56:55,402 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/C/d6be2ed31c414b00a6f5ba349909f7f2 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/d6be2ed31c414b00a6f5ba349909f7f2 2023-03-20 10:56:55,420 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:55,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42986 deadline: 1679309875418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,424 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/d6be2ed31c414b00a6f5ba349909f7f2, entries=150, sequenceid=111, filesize=11.6 K 2023-03-20 10:56:55,425 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 9fc4d04a6b2183ce1217c4847d39aded in 997ms, sequenceid=111, compaction requested=true 2023-03-20 10:56:55,426 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9fc4d04a6b2183ce1217c4847d39aded: 2023-03-20 10:56:55,426 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:55,426 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:55,426 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:55,426 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,426 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:55,426 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:55,427 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:55,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:55,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:55,428 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:55,428 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:55,428 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:55,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:55,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:55,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:55,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:55,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47cfb602[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:55,430 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:55,430 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:55,430 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:55,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:55,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:55,431 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:55,431 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:55,431 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:55,431 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,431 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:55,431 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,431 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:55,431 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:55,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:55,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:55,432 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:55,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:55,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:55,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:55,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:55,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:55,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:55,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:55,437 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:55,437 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:55,437 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:55,437 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:55,437 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:55,452 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:55,452 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:55,452 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,452 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:55,453 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:55,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:55,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:55,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:55,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:55,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:55,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:55,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:55,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:55,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:55,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:55,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:55,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,460 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:55,460 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:55,460 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:55,460 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:55,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:55,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:55,460 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:55,460 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:55,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:55,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:55,460 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,460 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:55,460 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:55,460 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:55,461 DEBUG [Thread-875] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:55,461 DEBUG [Thread-875] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:55,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:55,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:55,462 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:56:55,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 9fc4d04a6b2183ce1217c4847d39aded 2023-03-20 10:56:55,473 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9fc4d04a6b2183ce1217c4847d39aded 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-20 10:56:55,474 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9fc4d04a6b2183ce1217c4847d39aded, store=A 2023-03-20 10:56:55,474 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-20 10:56:55,474 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9fc4d04a6b2183ce1217c4847d39aded, store=B 2023-03-20 10:56:55,474 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-20 10:56:55,474 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9fc4d04a6b2183ce1217c4847d39aded, store=C 2023-03-20 10:56:55,474 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-20 10:56:55,507 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:55,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42936 deadline: 1679309875503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,512 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:55,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42958 deadline: 1679309875507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,522 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303208164828578aa4e8fb2af26d8eb39a2b0 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303208164828578aa4e8fb2af26d8eb39a2b0 2023-03-20 10:56:55,536 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=149, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/A/f62fd68cef9c44e4a448d7317e332f61 2023-03-20 10:56:55,561 DEBUG [Thread-875] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:55,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:55,565 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=149 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/B/b6f5a113e7714d24988a7c0e862019c0 2023-03-20 10:56:55,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:55,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:55,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:55,567 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:55,568 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:55,568 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:55,568 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:55,568 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:55,568 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:55,569 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:55,569 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:55,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:55,570 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,570 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:55,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:55,570 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:55,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:55,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:55,571 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:55,578 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:55,578 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:55,578 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:55,578 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:55,578 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:55,585 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:55,585 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:55,585 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:55,585 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. started... 2023-03-20 10:56:55,585 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. as already flushing 2023-03-20 10:56:55,585 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:55,585 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:55,585 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:55,585 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,585 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:55,585 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:55,593 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:55,593 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:55,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:55,594 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:55,594 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:55,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:55,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:55,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:55,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:55,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:55,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c0b2000[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:55,596 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:55,596 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:55,598 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:55,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:55,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:55,598 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,598 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:55,598 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:55,599 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:55,599 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:55,600 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,600 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:55,600 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:55,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:55,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:55,600 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:55,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:55,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:55,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:55,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:55,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:55,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:55,602 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:55,602 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:55,602 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:55,602 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:55,603 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:55,603 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:55,604 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:55,604 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:55,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:55,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:55,604 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:55,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:55,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:55,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:55,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:55,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:55,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:55,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:55,605 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:55,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:55,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:55,606 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:55,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:55,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,608 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:55,608 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,608 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:55,608 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:55,608 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,608 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:55,608 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:55,608 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:55,608 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:55,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:55,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:55,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:55,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:55,609 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:56:55,611 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:55,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42936 deadline: 1679309875608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,613 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:55,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42958 deadline: 1679309875613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,614 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:55,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:55,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:55,614 DEBUG [Thread-875] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:55,614 DEBUG [Thread-875] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:55,714 DEBUG [Thread-875] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:55,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:55,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:55,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:55,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:55,722 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:55,722 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:55,722 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:55,722 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:55,723 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:55,723 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:55,724 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:55,724 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:55,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:55,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:55,725 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:55,725 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:55,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42986 deadline: 1679309875723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:55,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:55,725 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:55,726 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:55,726 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:55,726 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:55,726 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:55,726 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:55,727 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:55,728 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:55,728 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:55,728 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. started... 2023-03-20 10:56:55,728 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. as already flushing 2023-03-20 10:56:55,728 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:55,728 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:55,728 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:55,728 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,728 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:55,728 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:55,729 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:55,729 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:55,729 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:55,730 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,730 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:55,730 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3be6842[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:55,730 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:55,730 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:55,730 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:55,731 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:55,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:55,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:55,731 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:55,731 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:55,732 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:55,732 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:55,733 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:55,733 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:55,733 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,733 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:55,733 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:55,733 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:55,733 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:55,733 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:55,733 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:55,733 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:55,735 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:55,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:55,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:55,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:55,735 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:55,735 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:55,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:55,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:55,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:55,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:55,736 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:55,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:55,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:55,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:55,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:55,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,738 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:55,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:55,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,740 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:55,740 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:55,740 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:55,740 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:55,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:55,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:55,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:55,740 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:55,740 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:55,740 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,740 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:55,741 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:55,741 DEBUG [Thread-875] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:55,741 DEBUG [Thread-875] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:55,741 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:55,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:55,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:55,815 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:55,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42936 deadline: 1679309875814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,816 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:55,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42958 deadline: 1679309875814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,841 DEBUG [Thread-875] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:55,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:55,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:55,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:55,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:55,848 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:55,848 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:55,848 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:55,848 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:55,849 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:55,849 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:55,850 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:55,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:55,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:55,851 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:55,857 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:55,857 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,858 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:55,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:55,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:55,858 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:55,865 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:55,865 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:55,865 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:55,865 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:55,865 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:55,865 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:55,865 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:55,866 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:55,866 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. started... 2023-03-20 10:56:55,866 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. as already flushing 2023-03-20 10:56:55,866 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:55,866 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:55,866 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:55,866 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,866 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:55,866 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:55,870 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,870 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:55,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:55,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:55,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:55,872 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:55,873 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:55,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:55,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:55,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:55,874 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:55,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30c4fd78[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:55,875 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:55,875 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:55,878 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:55,878 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,878 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:55,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:55,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:55,881 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:55,881 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:55,881 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:55,885 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:55,885 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:55,885 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:55,885 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:55,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:55,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:55,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:55,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:55,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:55,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:55,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:55,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:55,890 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:55,890 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:55,890 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:55,890 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:55,890 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:55,893 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:55,893 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:55,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:55,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:55,893 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:55,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:55,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:55,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:55,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:55,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:55,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:55,894 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:55,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:55,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:55,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:55,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,895 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:55,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:55,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,898 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,898 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:55,898 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:55,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:55,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:55,898 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:55,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:55,898 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:55,898 DEBUG [Thread-875] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:55,898 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:55,898 DEBUG [Thread-875] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:55,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:55,898 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:55,898 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:55,898 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:55,898 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:55,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:55,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:55,997 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=149 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/C/ee90dee8ab6e4b9b9d080d19cffcec42 2023-03-20 10:56:55,998 DEBUG [Thread-875] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:55,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:56,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:56,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:56,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:56,001 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:56,001 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:56,001 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/A/f62fd68cef9c44e4a448d7317e332f61 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/f62fd68cef9c44e4a448d7317e332f61 2023-03-20 10:56:56,001 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:56,002 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:56,002 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:56,002 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:56,002 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:56,002 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:56,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:56,003 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,003 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:56,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:56,003 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:56,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:56,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:56,003 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:56,004 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:56,004 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:56,004 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:56,004 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:56,004 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:56,004 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:56,004 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:56,004 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:56,004 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. started... 2023-03-20 10:56:56,004 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. as already flushing 2023-03-20 10:56:56,004 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:56,004 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:56,004 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:56,004 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,004 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:56,004 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:56,005 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:56,005 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:56,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:56,006 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:56,006 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:56,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:56,006 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/f62fd68cef9c44e4a448d7317e332f61, entries=150, sequenceid=149, filesize=25.3 K 2023-03-20 10:56:56,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:56,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:56,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,007 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/B/b6f5a113e7714d24988a7c0e862019c0 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/b6f5a113e7714d24988a7c0e862019c0 2023-03-20 10:56:56,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:56,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:56,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7673ef29[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:56,007 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:56,007 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:56,008 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:56,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:56,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:56,008 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:56,008 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:56,008 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:56,008 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,008 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:56,010 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,010 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:56,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:56,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:56,010 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:56,010 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:56,010 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/b6f5a113e7714d24988a7c0e862019c0, entries=150, sequenceid=149, filesize=11.8 K 2023-03-20 10:56:56,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:56,011 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/C/ee90dee8ab6e4b9b9d080d19cffcec42 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/ee90dee8ab6e4b9b9d080d19cffcec42 2023-03-20 10:56:56,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:56,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:56,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:56,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:56,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:56,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:56,013 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:56,013 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:56,013 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:56,013 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:56,013 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:56,015 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/ee90dee8ab6e4b9b9d080d19cffcec42, entries=150, sequenceid=149, filesize=11.8 K 2023-03-20 10:56:56,015 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~405.56 KB/415296, currentSize=40.25 KB/41220 for 9fc4d04a6b2183ce1217c4847d39aded in 543ms, sequenceid=149, compaction requested=true 2023-03-20 10:56:56,015 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9fc4d04a6b2183ce1217c4847d39aded: 2023-03-20 10:56:56,015 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:56:56,015 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:56:56,015 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 10:56:56,016 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:56:56,015 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 10:56:56,017 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 103237 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 10:56:56,017 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1912): 9fc4d04a6b2183ce1217c4847d39aded/A is initiating minor compaction (all files) 2023-03-20 10:56:56,017 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9fc4d04a6b2183ce1217c4847d39aded/A in TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:56,017 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/9c31b8bb7b964321af7a3300a48b0852, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/df09ecfeda5f41758eca14656efe0490, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/68564e81a07c46789d63bd6b483206a0, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/f62fd68cef9c44e4a448d7317e332f61] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp, totalSize=100.8 K 2023-03-20 10:56:56,017 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47797 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 10:56:56,017 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 9c31b8bb7b964321af7a3300a48b0852, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1679309810155 2023-03-20 10:56:56,017 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): 9fc4d04a6b2183ce1217c4847d39aded/B is initiating minor compaction (all files) 2023-03-20 10:56:56,017 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9fc4d04a6b2183ce1217c4847d39aded/B in TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:56,017 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/6a68e3e657294e52a8fcc770a5f7df9b, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/be09314619df44d3832029fe8762e5c6, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/4be62bbe615a418296b8f54c7e4ddc16, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/b6f5a113e7714d24988a7c0e862019c0] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp, totalSize=46.7 K 2023-03-20 10:56:56,017 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting df09ecfeda5f41758eca14656efe0490, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1679309810909 2023-03-20 10:56:56,018 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 6a68e3e657294e52a8fcc770a5f7df9b, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1679309810155 2023-03-20 10:56:56,018 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:56,018 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:56,018 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:56,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:56,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:56,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:56,018 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting be09314619df44d3832029fe8762e5c6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1679309810909 2023-03-20 10:56:56,018 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 68564e81a07c46789d63bd6b483206a0, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=111, earliestPutTs=1679309812984 2023-03-20 10:56:56,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:56,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:56,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:56,019 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:56,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:56,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:56,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:56,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:56,019 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 4be62bbe615a418296b8f54c7e4ddc16, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=111, earliestPutTs=1679309812984 2023-03-20 10:56:56,019 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting f62fd68cef9c44e4a448d7317e332f61, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1679309815117 2023-03-20 10:56:56,020 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting b6f5a113e7714d24988a7c0e862019c0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1679309815117 2023-03-20 10:56:56,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:56,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,026 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:56,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:56,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,029 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:56,029 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:56,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:56,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:56,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:56,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:56,030 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,030 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:56,030 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:56,030 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,030 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:56,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:56,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:56,030 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:56,030 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:56,030 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:56,030 DEBUG [Thread-875] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:56,030 DEBUG [Thread-875] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:56,036 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): 9fc4d04a6b2183ce1217c4847d39aded#B#compaction#132 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:56:56,058 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9fc4d04a6b2183ce1217c4847d39aded#A#compaction#133 average throughput is 0.99 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:56:56,062 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/B/ae4c387285774dccaacee853fe0e2362 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/ae4c387285774dccaacee853fe0e2362 2023-03-20 10:56:56,069 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9fc4d04a6b2183ce1217c4847d39aded/B of 9fc4d04a6b2183ce1217c4847d39aded into ae4c387285774dccaacee853fe0e2362(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:56:56,069 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9fc4d04a6b2183ce1217c4847d39aded: 2023-03-20 10:56:56,069 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded., storeName=9fc4d04a6b2183ce1217c4847d39aded/B, priority=12, startTime=1679309816015; duration=0sec 2023-03-20 10:56:56,069 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:56:56,069 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 10:56:56,071 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47797 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 10:56:56,072 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): 9fc4d04a6b2183ce1217c4847d39aded/C is initiating minor compaction (all files) 2023-03-20 10:56:56,072 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9fc4d04a6b2183ce1217c4847d39aded/C in TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:56,072 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/c719bc090a4a49ed9f7725a2c3f864ab, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/5a3487b49be6401a8ae673f5b5660673, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/d6be2ed31c414b00a6f5ba349909f7f2, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/ee90dee8ab6e4b9b9d080d19cffcec42] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp, totalSize=46.7 K 2023-03-20 10:56:56,072 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting c719bc090a4a49ed9f7725a2c3f864ab, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1679309810155 2023-03-20 10:56:56,072 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 5a3487b49be6401a8ae673f5b5660673, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1679309810909 2023-03-20 10:56:56,073 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting d6be2ed31c414b00a6f5ba349909f7f2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=111, earliestPutTs=1679309812984 2023-03-20 10:56:56,073 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting ee90dee8ab6e4b9b9d080d19cffcec42, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1679309815117 2023-03-20 10:56:56,092 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): 9fc4d04a6b2183ce1217c4847d39aded#C#compaction#134 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:56:56,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 9fc4d04a6b2183ce1217c4847d39aded 2023-03-20 10:56:56,119 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9fc4d04a6b2183ce1217c4847d39aded 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-20 10:56:56,119 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9fc4d04a6b2183ce1217c4847d39aded, store=A 2023-03-20 10:56:56,119 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:56,119 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9fc4d04a6b2183ce1217c4847d39aded, store=B 2023-03-20 10:56:56,119 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:56,120 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9fc4d04a6b2183ce1217c4847d39aded, store=C 2023-03-20 10:56:56,120 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:56,127 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/C/7725125e02a840699e78029a30adff1d as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/7725125e02a840699e78029a30adff1d 2023-03-20 10:56:56,130 DEBUG [Thread-875] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:56,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:56,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:56,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:56,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:56,139 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:56,140 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:56,140 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:56,140 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:56,140 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:56,140 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:56,141 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,141 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:56,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:56,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:56,141 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,141 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:56,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:56,141 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:56,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:56,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:56,141 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:56,144 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9fc4d04a6b2183ce1217c4847d39aded/C of 9fc4d04a6b2183ce1217c4847d39aded into 7725125e02a840699e78029a30adff1d(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:56:56,144 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9fc4d04a6b2183ce1217c4847d39aded: 2023-03-20 10:56:56,144 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded., storeName=9fc4d04a6b2183ce1217c4847d39aded/C, priority=12, startTime=1679309816016; duration=0sec 2023-03-20 10:56:56,144 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:56:56,148 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:56,148 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:56,148 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:56,148 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:56,148 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:56,148 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:56,148 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:56,148 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:56,148 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. started... 2023-03-20 10:56:56,148 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. as already flushing 2023-03-20 10:56:56,148 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:56,148 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:56,148 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:56,148 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,149 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:56,149 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:56,152 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:56,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:56,152 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:56,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:56,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:56,152 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:56,153 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:56,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:56,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:56,153 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:56,153 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c362843[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:56,153 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:56,153 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:56,154 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:56,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:56,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:56,154 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:56,154 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:56,154 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:56,154 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,154 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:56,155 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,155 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:56,155 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:56,155 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:56,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:56,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:56,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:56,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:56,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:56,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:56,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:56,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:56,158 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:56,158 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:56,158 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:56,158 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:56,158 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:56,158 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:56,159 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:56,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:56,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:56,159 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:56,159 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:56,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:56,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:56,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:56,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:56,160 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:56,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:56,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:56,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:56,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:56,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:56,161 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:56,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:56,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,163 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:56,163 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:56,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:56,163 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:56,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:56,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:56,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:56,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:56,164 DEBUG [Thread-875] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:56,164 DEBUG [Thread-875] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:56,164 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:56:56,163 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:56,163 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:56,164 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:56,164 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,164 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:56,164 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:56,164 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:56,177 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679309816163, max timestamp=1679309816175Num uniques 859; 2023-03-20 10:56:56,177 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-20 10:56:56,178 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679309816163, max timestamp=1679309816175Num uniques 859; 2023-03-20 10:56:56,178 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-20 10:56:56,211 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:56,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42958 deadline: 1679309876208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,211 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:56,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42936 deadline: 1679309876209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,219 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:56,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43002 deadline: 1679309876214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,222 DEBUG [Thread-871] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8202 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded., hostname=jenkins-hbase20.apache.org,36041,1679309777062, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-20 10:56:56,238 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:56,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42986 deadline: 1679309876237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,264 DEBUG [Thread-875] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:56,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:56,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:56,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:56,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:56,273 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:56,273 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:56,273 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:56,273 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:56,274 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:56,274 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:56,275 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:56,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:56,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:56,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:56,276 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:56,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:56,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:56,276 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:56,277 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,277 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,277 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:56,286 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:56,286 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:56,286 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:56,286 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:56,286 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:56,290 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:56,290 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:56,290 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:56,290 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. started... 2023-03-20 10:56:56,290 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. as already flushing 2023-03-20 10:56:56,291 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:56,291 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:56,291 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:56,291 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,291 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:56,291 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:56,295 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:56,295 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:56,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:56,295 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:56,295 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:56,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:56,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:56,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:56,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:56,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:56,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69181ae5[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:56,300 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:56,300 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:56,302 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:56,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:56,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:56,302 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:56,302 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:56,302 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:56,303 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,303 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:56,303 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,304 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:56,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:56,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:56,304 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:56,304 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:56,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:56,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:56,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:56,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:56,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:56,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:56,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:56,306 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:56,306 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:56,306 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:56,306 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:56,306 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:56,316 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:56,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42958 deadline: 1679309876313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,316 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:56,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42936 deadline: 1679309876314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,318 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:56,318 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:56,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:56,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:56,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:56,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:56,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:56,320 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:56,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:56,320 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:56,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:56,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:56,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:56,322 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:56,327 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,327 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:56,327 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:56,327 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,327 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:56,327 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:56,327 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:56,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:56,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:56,329 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:56:56,330 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:56,330 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:56,329 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:56,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:56,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:56,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:56,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:56,331 DEBUG [Thread-875] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:56,332 DEBUG [Thread-875] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:56,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:56,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:56,432 DEBUG [Thread-875] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:56,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:56,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:56,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:56,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:56,437 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:56,437 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:56,437 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:56,437 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:56,442 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:56,442 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:56,443 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:56,443 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:56,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:56,443 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,443 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:56,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:56,444 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:56,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:56,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:56,444 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:56,458 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:56,459 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:56,459 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:56,459 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:56,459 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:56,462 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:56,462 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:56,462 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:56,462 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. started... 2023-03-20 10:56:56,462 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. as already flushing 2023-03-20 10:56:56,462 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:56,462 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:56,462 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:56,462 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,462 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:56,462 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:56,464 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:56,464 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:56,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:56,465 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:56,465 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:56,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:56,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:56,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:56,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:56,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:56,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ea01583[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:56,466 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:56,467 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:56,467 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:56,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:56,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:56,468 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:56,468 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:56,468 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:56,468 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,469 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:56,469 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:56,469 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:56,469 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:56,469 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:56,469 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:56,469 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:56,469 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:56,469 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:56,469 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:56,470 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:56,470 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:56,470 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:56,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:56,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:56,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:56,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:56,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:56,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:56,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:56,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:56,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:56,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:56,472 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:56,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:56,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,473 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:56,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:56,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,477 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,477 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:56,477 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:56,477 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,477 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:56,478 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:56,478 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:56,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:56,478 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:56,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:56,478 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:56,478 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:56,478 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:56,478 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:56,478 DEBUG [Thread-875] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:56,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:56,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:56,478 DEBUG [Thread-875] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:56,526 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:56,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42936 deadline: 1679309876517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,526 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:56,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42958 deadline: 1679309876519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,544 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303207f4d9bca78a94b52a44dd97dc3a721f4 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303207f4d9bca78a94b52a44dd97dc3a721f4 2023-03-20 10:56:56,546 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/A/dc6177d1b44449119f504531a2618b47 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/dc6177d1b44449119f504531a2618b47 2023-03-20 10:56:56,551 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=161, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/A/89fe1a698b064675a497fefe7775f72e 2023-03-20 10:56:56,553 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9fc4d04a6b2183ce1217c4847d39aded/A of 9fc4d04a6b2183ce1217c4847d39aded into dc6177d1b44449119f504531a2618b47(size=25.5 K), total size for store is 25.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:56:56,553 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9fc4d04a6b2183ce1217c4847d39aded: 2023-03-20 10:56:56,553 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded., storeName=9fc4d04a6b2183ce1217c4847d39aded/A, priority=12, startTime=1679309816015; duration=0sec 2023-03-20 10:56:56,553 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:56:56,578 DEBUG [Thread-875] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:56,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:56,580 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:56,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:56,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:56,581 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:56,581 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:56,581 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:56,581 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:56,581 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:56,582 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:56,582 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,582 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:56,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:56,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:56,582 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,582 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:56,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:56,583 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:56,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:56,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:56,583 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:56,583 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:56,583 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:56,583 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:56,583 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:56,583 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:56,583 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:56,583 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:56,583 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:56,583 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. started... 2023-03-20 10:56:56,584 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. as already flushing 2023-03-20 10:56:56,584 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:56,584 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:56,584 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:56,584 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,584 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:56,584 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:56,584 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,585 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:56,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:56,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:56,585 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:56,585 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:56,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:56,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:56,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:56,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:56,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:56,586 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:56,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17c3b7d9[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:56,586 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:56,587 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:56,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:56,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:56,587 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:56,587 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,587 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:56,587 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:56,587 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:56,588 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,588 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:56,588 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:56,588 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:56,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:56,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:56,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:56,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:56,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:56,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:56,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:56,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:56,591 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:56,591 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:56,591 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:56,591 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:56,591 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:56,591 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:56,602 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:56,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:56,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:56,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:56,602 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:56,602 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:56,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:56,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:56,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:56,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:56,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:56,603 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:56,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:56,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:56,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:56,604 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:56,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:56,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,610 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,610 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:56,610 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:56,610 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,610 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:56,610 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:56,610 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:56,610 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:56,610 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:56,610 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:56,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:56,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:56,610 DEBUG [Thread-875] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:56,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:56,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:56,610 DEBUG [Thread-875] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:56,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:56,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:56,710 DEBUG [Thread-875] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:56,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:56,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:56,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:56,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:56,719 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:56,719 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:56,719 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:56,719 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:56,720 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:56,720 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:56,720 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,720 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:56,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:56,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:56,721 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:56,721 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:56,721 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:56,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:56,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:56,722 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:56,722 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:56,722 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:56,722 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:56,722 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:56,722 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:56,727 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:56,727 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:56,727 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:56,727 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. started... 2023-03-20 10:56:56,728 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. as already flushing 2023-03-20 10:56:56,728 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:56,728 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:56,728 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:56,728 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,728 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:56,728 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:56,735 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,735 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:56,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:56,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:56,736 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:56,736 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:56,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:56,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:56,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:56,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:56,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:56,737 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:56,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6acaf152[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:56,737 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:56,738 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,738 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:56,742 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:56,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:56,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:56,742 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:56,742 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:56,742 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:56,743 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,743 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:56,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:56,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:56,743 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:56,743 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:56,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:56,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:56,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:56,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:56,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:56,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:56,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:56,745 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:56,745 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:56,745 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:56,745 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:56,745 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:56,746 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:56,746 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:56,746 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:56,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:56,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:56,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:56,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:56,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:56,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:56,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:56,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:56,747 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:56,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:56,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:56,748 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:56,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:56,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:56,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,751 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,751 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:56,751 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:56,751 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:56,751 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,751 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:56,751 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:56,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:56,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:56,751 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:56,751 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:56,751 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:56,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:56,751 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:56:56,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:56,751 DEBUG [Thread-875] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:56,751 DEBUG [Thread-875] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:56,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:56,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:56,829 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:56,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42936 deadline: 1679309876829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,833 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:56,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42958 deadline: 1679309876830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,851 DEBUG [Thread-875] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:56,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:56,863 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:56,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:56,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:56,870 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:56,870 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:56,870 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:56,870 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:56,870 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:56,870 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:56,871 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,871 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:56,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:56,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:56,871 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,871 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:56,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:56,872 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:56,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:56,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:56,872 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:56,873 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:56,873 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:56,873 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:56,873 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:56,873 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:56,874 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:56,874 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:56,874 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:56,874 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. started... 2023-03-20 10:56:56,874 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. as already flushing 2023-03-20 10:56:56,874 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:56,874 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:56,874 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:56,874 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,874 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:56,874 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:56,875 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,875 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:56,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:56,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:56,875 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:56,875 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:56,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:56,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:56,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:56,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:56,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:56,876 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:56,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@432c01dc[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:56,876 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:56,877 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:56,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:56,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:56,877 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:56,877 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:56,877 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,877 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:56,877 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:56,878 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,878 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:56,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:56,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:56,878 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:56,878 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:56,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:56,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:56,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:56,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:56,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:56,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:56,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:56,880 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:56,880 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:56,880 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:56,880 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:56,880 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:56,881 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:56,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:56,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:56,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:56,881 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:56,881 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:56,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:56,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:56,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:56,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:56,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:56,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:56,882 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:56,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:56,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:56,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,882 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:56,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:56,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,895 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,895 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:56,895 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:56,895 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:56,895 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:56,895 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:56,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:56,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:56,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:56,895 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:56,895 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:56,895 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:56,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:56,895 DEBUG [Thread-875] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:56,895 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:56,895 DEBUG [Thread-875] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:56,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:56,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:56,966 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=161 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/B/f17638332af0452aa0ff703c7c1828a9 2023-03-20 10:56:56,996 DEBUG [Thread-875] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:56,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:56,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:56,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:56,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:56,999 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:56,999 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:56,999 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:56,999 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:56,999 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:56,999 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:57,000 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:57,000 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:57,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:57,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:57,000 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:57,000 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:57,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:57,001 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:57,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:57,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:57,001 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:57,001 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:57,001 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:57,001 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:57,002 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:57,002 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:57,002 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:57,002 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:57,002 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:57,002 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. started... 2023-03-20 10:56:57,002 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. as already flushing 2023-03-20 10:56:57,002 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:57,002 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:57,002 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:57,002 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:57,002 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:57,002 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:57,003 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:57,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:57,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:57,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:57,003 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:57,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:57,004 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:57,004 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:57,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:57,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:57,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:57,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:57,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:57,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d5e0837[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:57,005 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:57,005 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:57,006 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:57,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:57,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:57,006 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:57,006 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:57,006 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:57,006 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:57,006 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:57,007 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:57,007 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:57,007 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:57,007 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:57,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:57,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:57,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:57,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:57,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:57,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:57,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:57,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:57,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:57,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:57,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:57,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:57,009 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:57,009 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:57,009 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:57,009 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:57,009 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:57,010 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:57,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:57,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:57,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:57,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:57,010 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:57,010 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:57,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:57,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:57,010 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:57,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:57,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:57,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:57,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:57,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:57,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:57,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:57,012 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:57,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:57,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:57,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:57,014 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:57,014 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:57,014 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:57,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:57,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:57,014 DEBUG [Thread-875] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:57,014 DEBUG [Thread-875] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:57,014 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:57,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:57,014 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:57,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:57,015 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:57,015 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:57,015 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:57,015 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:57,015 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:57,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:57,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:57,115 DEBUG [Thread-875] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:57,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:57,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:57,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:57,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:57,126 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:57,126 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:57,126 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:57,126 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:57,127 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:57,127 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:57,128 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:57,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:57,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:57,128 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:57,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:57,128 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:57,128 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:57,129 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:57,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:57,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:57,129 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:57,129 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:57,129 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:57,129 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:57,129 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:57,129 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:57,134 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:57,134 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:57,134 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:57,134 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. started... 2023-03-20 10:56:57,134 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. as already flushing 2023-03-20 10:56:57,134 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:57,134 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:57,134 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:57,134 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:57,134 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:57,134 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:57,154 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:57,154 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:57,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:57,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:57,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:57,154 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:57,155 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:57,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:57,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:57,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:57,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:57,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:57,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:57,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37290d3a[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:57,156 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:57,156 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:57,162 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:57,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:57,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:57,162 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:57,162 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:57,162 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:57,162 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:57,162 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:57,163 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:57,163 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:57,163 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:57,163 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:57,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:57,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:57,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:57,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:57,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:57,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:57,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:57,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:57,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:57,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:57,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:57,167 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:57,167 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:57,167 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:57,167 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:57,167 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:57,167 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:57,169 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:57,169 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:57,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:57,169 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:57,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:57,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:57,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:57,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:57,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:57,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:57,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:57,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:57,171 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:57,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:57,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:57,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:57,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:57,173 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:57,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:57,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:57,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:57,179 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:57,179 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:57,179 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:57,179 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:57,179 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:57,179 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:57,179 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:57,180 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:57,180 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:57,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:57,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:57,180 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:57,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:57,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:57,180 DEBUG [Thread-875] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:57,180 DEBUG [Thread-875] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:57,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:57,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:57,185 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:57,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42972 deadline: 1679309877185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:57,186 DEBUG [Thread-865] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4159 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded., hostname=jenkins-hbase20.apache.org,36041,1679309777062, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-20 10:56:57,248 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:57,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42986 deadline: 1679309877247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:57,280 DEBUG [Thread-875] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:57,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:57,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:57,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:57,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:57,299 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:57,299 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:57,299 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:57,299 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:57,300 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:57,300 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:57,303 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:57,303 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:57,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:57,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:57,304 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:57,304 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:57,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:57,304 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:57,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:57,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:57,304 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:57,305 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:57,305 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:57,305 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:57,305 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:57,305 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:57,305 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:57,305 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:57,305 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:57,305 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. started... 2023-03-20 10:56:57,305 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. as already flushing 2023-03-20 10:56:57,305 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:57,305 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:57,305 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:57,305 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:57,306 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:57,306 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:57,306 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:57,307 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:57,307 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:57,307 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:57,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:57,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:57,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:57,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:57,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:57,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:57,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:57,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:57,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:57,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a542aec[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:57,316 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:57,316 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:57,321 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:57,322 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:57,322 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:57,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:57,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:57,322 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:57,322 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:57,322 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:57,325 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:57,325 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:57,325 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:57,325 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:57,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:57,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:57,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:57,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:57,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:57,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:57,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:57,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:57,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:57,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:57,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:57,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:57,329 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:57,329 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:57,329 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:57,329 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:57,329 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:57,337 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:57,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:57,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:57,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:57,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:57,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:57,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:57,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:57,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:57,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:57,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:57,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:57,342 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:57,342 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:57,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:57,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:57,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:57,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:57,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:57,345 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:57,345 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:57,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42936 deadline: 1679309877342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:57,346 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:57,346 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:57,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42958 deadline: 1679309877344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:57,348 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:57,348 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:57,348 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:57,348 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:57,348 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:57,348 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:57,348 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:57,348 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:57,348 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:57,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:57,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:57,348 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:57,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:57,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:57,349 DEBUG [Thread-875] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:57,349 DEBUG [Thread-875] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:57,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:57,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:57,390 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=161 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/C/329cfc79a9a84b808f795f9f880cdded 2023-03-20 10:56:57,401 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/A/89fe1a698b064675a497fefe7775f72e as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/89fe1a698b064675a497fefe7775f72e 2023-03-20 10:56:57,411 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/89fe1a698b064675a497fefe7775f72e, entries=200, sequenceid=161, filesize=32.1 K 2023-03-20 10:56:57,412 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/B/f17638332af0452aa0ff703c7c1828a9 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/f17638332af0452aa0ff703c7c1828a9 2023-03-20 10:56:57,418 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/f17638332af0452aa0ff703c7c1828a9, entries=150, sequenceid=161, filesize=11.8 K 2023-03-20 10:56:57,421 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/C/329cfc79a9a84b808f795f9f880cdded as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/329cfc79a9a84b808f795f9f880cdded 2023-03-20 10:56:57,435 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/329cfc79a9a84b808f795f9f880cdded, entries=150, sequenceid=161, filesize=11.8 K 2023-03-20 10:56:57,436 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=125.23 KB/128240 for 9fc4d04a6b2183ce1217c4847d39aded in 1317ms, sequenceid=161, compaction requested=false 2023-03-20 10:56:57,436 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9fc4d04a6b2183ce1217c4847d39aded: 2023-03-20 10:56:57,449 DEBUG [Thread-875] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:57,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:57,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:57,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:57,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:57,453 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:57,453 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:57,453 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:57,453 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:57,453 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:57,453 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:57,455 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:57,455 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:57,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:57,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:57,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:57,456 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:57,456 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:57,456 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:57,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:57,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:57,457 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:57,458 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:57,458 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:57,458 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:57,458 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:57,458 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:57,459 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:57,459 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:57,460 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:57,460 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. started... 2023-03-20 10:56:57,460 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9fc4d04a6b2183ce1217c4847d39aded 3/3 column families, dataSize=125.23 KB heapSize=354.02 KB 2023-03-20 10:56:57,460 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9fc4d04a6b2183ce1217c4847d39aded, store=A 2023-03-20 10:56:57,460 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:56:57,460 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9fc4d04a6b2183ce1217c4847d39aded, store=B 2023-03-20 10:56:57,460 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:56:57,460 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9fc4d04a6b2183ce1217c4847d39aded, store=C 2023-03-20 10:56:57,460 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:56:57,919 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032043bd71e1a4b44ed7865482f7cc6a5864 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032043bd71e1a4b44ed7865482f7cc6a5864 2023-03-20 10:56:57,968 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=192, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/A/9c4d50edf211489fb2e8c8d9ca86e846 2023-03-20 10:56:58,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 9fc4d04a6b2183ce1217c4847d39aded 2023-03-20 10:56:58,358 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. as already flushing 2023-03-20 10:56:58,379 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:58,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42936 deadline: 1679309878375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:58,379 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:58,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42958 deadline: 1679309878377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:58,409 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=192 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/B/e5e7adf94554422f908c7f6fb30dff3a 2023-03-20 10:56:58,469 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=58.14 KB at sequenceid=192 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/C/9bdb3eb481744df19b69cf78e641b60e 2023-03-20 10:56:58,484 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:58,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42958 deadline: 1679309878482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:58,485 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:58,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42936 deadline: 1679309878482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:58,487 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/A/9c4d50edf211489fb2e8c8d9ca86e846 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/9c4d50edf211489fb2e8c8d9ca86e846 2023-03-20 10:56:58,497 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/9c4d50edf211489fb2e8c8d9ca86e846, entries=150, sequenceid=192, filesize=25.3 K 2023-03-20 10:56:58,502 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/B/e5e7adf94554422f908c7f6fb30dff3a as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/e5e7adf94554422f908c7f6fb30dff3a 2023-03-20 10:56:58,507 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/e5e7adf94554422f908c7f6fb30dff3a, entries=150, sequenceid=192, filesize=11.8 K 2023-03-20 10:56:58,508 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/C/9bdb3eb481744df19b69cf78e641b60e as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/9bdb3eb481744df19b69cf78e641b60e 2023-03-20 10:56:58,516 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/9bdb3eb481744df19b69cf78e641b60e, entries=150, sequenceid=192, filesize=11.8 K 2023-03-20 10:56:58,517 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~125.23 KB/128240, heapSize ~353.97 KB/362464, currentSize=60.38 KB/61830 for 9fc4d04a6b2183ce1217c4847d39aded in 1057ms, sequenceid=192, compaction requested=true 2023-03-20 10:56:58,517 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9fc4d04a6b2183ce1217c4847d39aded: 2023-03-20 10:56:58,517 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:58,517 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:56:58,517 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:56:58,517 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:58,522 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:56:58,522 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:56:58,523 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:58,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:58,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:58,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:58,523 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:58,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:58,524 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:58,524 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:56:58,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:58,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:58,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:58,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:58,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:56:58,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2955e89b[Count = 0] remaining members to acquire global barrier 2023-03-20 10:56:58,525 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:56:58,525 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:58,526 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:58,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:58,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:58,526 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:56:58,526 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:56:58,526 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:56:58,526 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:58,526 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:56:58,530 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:58,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:58,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:58,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:58,531 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:56:58,531 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:58,531 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:56:58,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:58,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:58,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:58,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:58,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:58,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:58,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:58,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:56:58,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:56:58,533 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:56:58,533 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:56:58,533 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:56:58,533 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:56:58,534 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:56:58,535 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:58,535 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:58,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:58,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:58,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:56:58,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:56:58,535 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:58,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:58,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:58,536 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:58,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:56:58,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:58,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:58,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:58,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:56:58,537 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:58,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:58,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:58,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:56:58,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:56:58,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:58,539 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:58,540 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:56:58,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:58,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:58,540 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:58,540 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:56:58,540 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:58,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:56:58,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:56:58,540 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:58,540 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:58,540 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:58,540 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:56:58,540 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:58,540 DEBUG [Thread-875] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:56:58,540 DEBUG [Thread-875] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:56:58,540 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:56:58,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:56:58,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:56:58,640 DEBUG [Thread-875] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:56:58,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:56:58,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:56:58,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:56:58,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:56:58,644 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:56:58,644 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:58,644 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:56:58,644 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:56:58,644 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:58,644 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:58,645 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:56:58,645 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:58,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:56:58,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:56:58,645 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:58,645 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:56:58,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:58,646 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:56:58,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:56:58,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:56:58,646 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:56:58,648 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:56:58,648 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:56:58,648 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:56:58,648 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:56:58,648 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:56:58,648 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:56:58,648 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:56:58,648 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:56:58,648 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. started... 2023-03-20 10:56:58,648 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9fc4d04a6b2183ce1217c4847d39aded 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-20 10:56:58,649 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9fc4d04a6b2183ce1217c4847d39aded, store=A 2023-03-20 10:56:58,649 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:58,649 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9fc4d04a6b2183ce1217c4847d39aded, store=B 2023-03-20 10:56:58,649 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:58,649 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9fc4d04a6b2183ce1217c4847d39aded, store=C 2023-03-20 10:56:58,649 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:56:58,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 9fc4d04a6b2183ce1217c4847d39aded 2023-03-20 10:56:58,688 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. as already flushing 2023-03-20 10:56:58,738 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1679309818725, max timestamp=1679309818735Num uniques 859; 2023-03-20 10:56:58,739 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-20 10:56:58,739 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679309818725, max timestamp=1679309818735Num uniques 859; 2023-03-20 10:56:58,739 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-20 10:56:58,739 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679309818725, max timestamp=1679309818735Num uniques 859; 2023-03-20 10:56:58,739 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-20 10:56:58,787 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:58,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42958 deadline: 1679309878779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:58,788 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:58,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42936 deadline: 1679309878785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:58,892 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:58,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42958 deadline: 1679309878889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:58,892 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:58,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42936 deadline: 1679309878890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:59,088 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303202d21f4f58caf4a09866f45a99afba288 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303202d21f4f58caf4a09866f45a99afba288 2023-03-20 10:56:59,102 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:59,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42936 deadline: 1679309879099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:59,103 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:59,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42958 deadline: 1679309879099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:59,107 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=204, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/A/6e630386549944c0bd784fcc3a6a120a 2023-03-20 10:56:59,255 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:59,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42986 deadline: 1679309879255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:59,257 DEBUG [Thread-867] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4150 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded., hostname=jenkins-hbase20.apache.org,36041,1679309777062, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-20 10:56:59,408 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:59,409 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:59,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42958 deadline: 1679309879407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:59,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42936 deadline: 1679309879406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:59,540 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=204 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/B/4ea0dfd6f05b42319bf06813fa611e2d 2023-03-20 10:56:59,915 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:59,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42958 deadline: 1679309879914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:59,921 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:56:59,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42936 deadline: 1679309879917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:56:59,981 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=204 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/C/b501de08c8b141b588706b7dfa423be2 2023-03-20 10:56:59,998 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/A/6e630386549944c0bd784fcc3a6a120a as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/6e630386549944c0bd784fcc3a6a120a 2023-03-20 10:57:00,009 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/6e630386549944c0bd784fcc3a6a120a, entries=150, sequenceid=204, filesize=25.3 K 2023-03-20 10:57:00,010 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/B/4ea0dfd6f05b42319bf06813fa611e2d as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/4ea0dfd6f05b42319bf06813fa611e2d 2023-03-20 10:57:00,016 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/4ea0dfd6f05b42319bf06813fa611e2d, entries=150, sequenceid=204, filesize=11.8 K 2023-03-20 10:57:00,017 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/C/b501de08c8b141b588706b7dfa423be2 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/b501de08c8b141b588706b7dfa423be2 2023-03-20 10:57:00,028 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/b501de08c8b141b588706b7dfa423be2, entries=150, sequenceid=204, filesize=11.8 K 2023-03-20 10:57:00,035 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for 9fc4d04a6b2183ce1217c4847d39aded in 1387ms, sequenceid=204, compaction requested=true 2023-03-20 10:57:00,035 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9fc4d04a6b2183ce1217c4847d39aded: 2023-03-20 10:57:00,035 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:57:00,035 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:00,035 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:00,036 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:00,036 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:00,036 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:00,037 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:00,038 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:00,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:00,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:00,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:00,038 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:00,038 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:00,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:00,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:00,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:00,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:00,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:00,040 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:00,040 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64874236[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:00,040 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:00,040 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:00,041 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:00,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:00,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:00,041 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:00,041 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:00,041 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:00,041 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:00,041 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:00,042 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:00,042 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:00,042 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:00,042 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:00,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:00,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:00,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:00,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:00,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:00,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:00,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:00,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:00,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:00,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:00,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:00,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:00,054 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:00,054 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:00,054 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:00,054 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:00,054 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:00,062 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:00,063 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:00,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:00,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:00,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:00,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:00,063 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:00,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:00,064 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:00,066 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:00,066 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:00,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:00,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:00,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:00,066 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:00,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:00,066 DEBUG [Thread-875] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:00,066 DEBUG [Thread-875] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:00,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:00,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:00,070 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:00,070 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:00,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:00,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:00,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:00,070 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:00,070 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:00,070 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:00,070 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:00,070 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:00,070 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:00,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:00,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:00,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:00,072 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:57:00,166 DEBUG [Thread-875] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:00,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:00,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:00,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:00,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:00,216 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:00,216 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:00,217 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:00,217 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:00,217 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:00,217 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:00,219 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:00,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:00,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:00,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:00,220 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:00,220 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:00,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:00,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:00,220 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:00,221 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:00,221 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:00,221 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:00,221 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:00,221 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:00,221 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:00,221 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:00,224 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:00,224 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:00,224 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:57:00,224 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. started... 2023-03-20 10:57:00,234 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9fc4d04a6b2183ce1217c4847d39aded 3/3 column families, dataSize=120.76 KB heapSize=340.45 KB 2023-03-20 10:57:00,234 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9fc4d04a6b2183ce1217c4847d39aded, store=A 2023-03-20 10:57:00,235 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:57:00,235 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9fc4d04a6b2183ce1217c4847d39aded, store=B 2023-03-20 10:57:00,235 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:57:00,235 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9fc4d04a6b2183ce1217c4847d39aded, store=C 2023-03-20 10:57:00,235 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:57:00,267 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032003108f7fce2944bd843f30e339ba6f8d to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032003108f7fce2944bd843f30e339ba6f8d 2023-03-20 10:57:00,700 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=238, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/A/c2dfa925447f4b4286a190f423e5ccf5 2023-03-20 10:57:00,733 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=238 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/B/4d80978249f54a3696f1e8d053a7603f 2023-03-20 10:57:00,785 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=238 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/C/99751ba587ef43dc8e1d7e873a1b721a 2023-03-20 10:57:00,801 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/A/c2dfa925447f4b4286a190f423e5ccf5 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/c2dfa925447f4b4286a190f423e5ccf5 2023-03-20 10:57:00,819 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/c2dfa925447f4b4286a190f423e5ccf5, entries=150, sequenceid=238, filesize=25.3 K 2023-03-20 10:57:00,823 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/B/4d80978249f54a3696f1e8d053a7603f as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/4d80978249f54a3696f1e8d053a7603f 2023-03-20 10:57:00,831 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/4d80978249f54a3696f1e8d053a7603f, entries=150, sequenceid=238, filesize=11.8 K 2023-03-20 10:57:00,832 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/C/99751ba587ef43dc8e1d7e873a1b721a as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/99751ba587ef43dc8e1d7e873a1b721a 2023-03-20 10:57:00,855 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/99751ba587ef43dc8e1d7e873a1b721a, entries=150, sequenceid=238, filesize=11.8 K 2023-03-20 10:57:00,856 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~340.41 KB/348576, currentSize=0 B/0 for 9fc4d04a6b2183ce1217c4847d39aded in 622ms, sequenceid=238, compaction requested=true 2023-03-20 10:57:00,856 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9fc4d04a6b2183ce1217c4847d39aded: 2023-03-20 10:57:00,856 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:57:00,857 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:00,857 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:00,857 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:00,857 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:00,857 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:00,858 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:00,858 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:00,858 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:00,858 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:00,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:00,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:00,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:00,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:00,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:00,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:00,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:00,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:00,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:00,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1bb8c341[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:00,861 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:00,861 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:00,861 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:00,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:00,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:00,862 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:00,862 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:00,862 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:00,862 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:00,862 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:00,863 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:00,863 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:00,863 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:00,863 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:00,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:00,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:00,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:00,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:00,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:00,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:00,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:00,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:00,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:00,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:00,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:00,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:00,866 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:00,866 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:00,866 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:00,866 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:00,866 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:00,867 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:00,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:00,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:00,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:00,867 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:00,867 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:00,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:00,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:00,867 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:00,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:00,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:00,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:00,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:00,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:00,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:00,869 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:00,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:00,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:00,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:00,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:00,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:00,870 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:00,871 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:00,871 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:00,871 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:00,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:00,871 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:00,871 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:00,871 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:00,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:00,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:00,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:00,871 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:00,871 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:00,871 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:00,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:00,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:00,871 DEBUG [Thread-875] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:00,871 DEBUG [Thread-875] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:00,871 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:57:00,971 DEBUG [Thread-875] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:00,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:00,981 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:00,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:00,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:00,984 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:00,984 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:00,984 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:00,984 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:00,986 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:00,986 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:00,987 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:00,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:00,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:00,987 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:00,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:00,988 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:00,988 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:00,988 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:00,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:00,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:00,989 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:00,996 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:00,996 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:00,996 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:00,996 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:00,996 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:00,996 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:00,996 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:00,996 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:57:00,996 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. started... 2023-03-20 10:57:00,996 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9fc4d04a6b2183ce1217c4847d39aded 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-03-20 10:57:00,997 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9fc4d04a6b2183ce1217c4847d39aded, store=A 2023-03-20 10:57:00,997 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:00,997 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9fc4d04a6b2183ce1217c4847d39aded, store=B 2023-03-20 10:57:00,997 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:00,997 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9fc4d04a6b2183ce1217c4847d39aded, store=C 2023-03-20 10:57:00,997 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:01,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 9fc4d04a6b2183ce1217c4847d39aded 2023-03-20 10:57:01,041 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. as already flushing 2023-03-20 10:57:01,103 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1679309821013, max timestamp=1679309821101Num uniques 859; 2023-03-20 10:57:01,103 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-03-20 10:57:01,104 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1679309821013, max timestamp=1679309821101Num uniques 859; 2023-03-20 10:57:01,104 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-03-20 10:57:01,105 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1679309821013, max timestamp=1679309821101Num uniques 859; 2023-03-20 10:57:01,105 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-03-20 10:57:01,157 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:01,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42958 deadline: 1679309881148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:01,160 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:01,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42936 deadline: 1679309881157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:01,213 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:01,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42972 deadline: 1679309881211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:01,218 DEBUG [Thread-865] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8187 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded., hostname=jenkins-hbase20.apache.org,36041,1679309777062, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-20 10:57:01,260 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:01,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42958 deadline: 1679309881259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:01,264 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:01,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42936 deadline: 1679309881261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:01,468 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230320c5219afa2269420496871b488fb6bc68 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320c5219afa2269420496871b488fb6bc68 2023-03-20 10:57:01,470 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:01,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42958 deadline: 1679309881463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:01,477 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:01,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42936 deadline: 1679309881474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:01,495 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=243, memsize=4.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/A/cb94e2f9fbb94325a40f6784aeb23e94 2023-03-20 10:57:01,525 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=243 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/B/2ff53b2e1b88479f93749e975bb249a8 2023-03-20 10:57:01,566 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=243 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/C/dd58f713e4954992a5600d4b2615b513 2023-03-20 10:57:01,576 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/A/cb94e2f9fbb94325a40f6784aeb23e94 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/cb94e2f9fbb94325a40f6784aeb23e94 2023-03-20 10:57:01,583 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/cb94e2f9fbb94325a40f6784aeb23e94, entries=100, sequenceid=243, filesize=18.5 K 2023-03-20 10:57:01,604 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/B/2ff53b2e1b88479f93749e975bb249a8 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/2ff53b2e1b88479f93749e975bb249a8 2023-03-20 10:57:01,622 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/2ff53b2e1b88479f93749e975bb249a8, entries=50, sequenceid=243, filesize=7.1 K 2023-03-20 10:57:01,623 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/C/dd58f713e4954992a5600d4b2615b513 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/dd58f713e4954992a5600d4b2615b513 2023-03-20 10:57:01,644 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/dd58f713e4954992a5600d4b2615b513, entries=50, sequenceid=243, filesize=7.1 K 2023-03-20 10:57:01,650 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~13.42 KB/13740, heapSize ~40.55 KB/41520, currentSize=187.85 KB/192360 for 9fc4d04a6b2183ce1217c4847d39aded in 654ms, sequenceid=243, compaction requested=true 2023-03-20 10:57:01,651 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9fc4d04a6b2183ce1217c4847d39aded: 2023-03-20 10:57:01,651 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:57:01,651 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:01,651 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:01,651 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:01,651 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:01,651 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:01,653 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:01,653 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:01,653 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:01,654 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:01,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:01,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:01,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:01,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:01,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:01,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:01,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:01,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:01,656 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:01,656 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30109990[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:01,656 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:01,656 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:01,657 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:01,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:01,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:01,657 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:01,657 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:01,657 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:01,658 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:01,658 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:01,658 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:01,658 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:01,658 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:01,659 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:01,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:01,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:01,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:01,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:01,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:01,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:01,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:01,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:01,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:01,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:01,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:01,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:01,661 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:01,661 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:01,662 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:01,662 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:01,662 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:01,663 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:01,663 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:01,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:01,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:01,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:01,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:01,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:01,665 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:01,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:01,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:01,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:01,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:01,666 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:01,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:01,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:01,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:01,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:01,668 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:01,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:01,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:01,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:01,670 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:01,670 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:01,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:01,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:01,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:01,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:01,671 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:01,671 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:01,671 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:01,671 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:01,671 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:01,671 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:01,671 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:01,671 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:01,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:01,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:01,678 DEBUG [Thread-875] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:01,678 DEBUG [Thread-875] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:01,780 DEBUG [Thread-875] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:01,781 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9fc4d04a6b2183ce1217c4847d39aded 3/3 column families, dataSize=194.56 KB heapSize=498.66 KB 2023-03-20 10:57:01,783 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9fc4d04a6b2183ce1217c4847d39aded, store=A 2023-03-20 10:57:01,783 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:57:01,783 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9fc4d04a6b2183ce1217c4847d39aded, store=B 2023-03-20 10:57:01,783 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:57:01,783 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9fc4d04a6b2183ce1217c4847d39aded, store=C 2023-03-20 10:57:01,783 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:57:01,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 9fc4d04a6b2183ce1217c4847d39aded 2023-03-20 10:57:01,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:01,798 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:01,799 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:01,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42958 deadline: 1679309881791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:01,800 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:01,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42936 deadline: 1679309881792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:01,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:01,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:01,842 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:01,842 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:01,842 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:01,842 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:01,849 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:01,849 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:01,855 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:01,856 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:01,856 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:01,858 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:01,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:01,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:01,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:01,859 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:01,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:01,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:01,859 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:01,878 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:01,878 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:01,878 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:01,878 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:01,878 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:01,886 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:01,886 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:01,890 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:57:01,890 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. started... 2023-03-20 10:57:01,890 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. as already flushing 2023-03-20 10:57:01,890 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:57:01,890 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:01,890 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:01,890 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:01,890 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:01,891 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:01,895 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:01,895 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:01,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:01,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:01,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:01,895 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:01,895 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:01,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:01,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:01,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:01,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:01,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:01,898 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:01,898 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3687be39[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:01,898 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:01,898 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:01,900 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:01,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:01,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:01,900 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:01,900 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:01,900 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:01,901 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:01,901 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:01,902 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:01,903 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:01,903 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:01,904 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:01,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:01,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:01,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:01,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:01,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:01,911 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:01,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42936 deadline: 1679309881901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:01,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:01,918 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:01,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42958 deadline: 1679309881906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:01,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:01,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:01,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:01,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:01,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:01,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:01,952 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:01,952 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:01,952 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:01,952 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:01,952 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:01,967 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:01,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:01,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:01,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:01,968 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:01,968 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:01,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:01,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:01,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:01,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:01,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:01,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:01,974 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:01,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:01,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:01,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:01,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:01,977 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:01,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:01,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:01,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:01,980 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:01,980 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:01,980 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:01,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:01,980 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:01,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:01,980 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:01,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:01,980 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:01,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:01,980 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:01,981 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:01,981 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:01,981 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:01,981 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:01,981 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:01,981 DEBUG [Thread-875] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:01,981 DEBUG [Thread-875] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:02,082 DEBUG [Thread-875] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:02,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:02,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:02,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:02,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:02,110 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:02,110 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:02,110 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:02,110 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:02,122 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:02,122 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:02,129 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:02,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42936 deadline: 1679309882119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,129 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:02,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42958 deadline: 1679309882126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,134 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:02,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:02,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:02,135 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:02,142 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,142 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:02,146 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:02,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:02,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:02,151 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:02,151 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:02,151 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:02,151 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:02,151 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:02,151 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:02,154 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:02,154 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:02,154 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:57:02,154 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. started... 2023-03-20 10:57:02,154 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. as already flushing 2023-03-20 10:57:02,154 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:57:02,154 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:02,154 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:02,154 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,154 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:02,155 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:02,156 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,156 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:02,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:02,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:02,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:02,156 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:02,156 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:02,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:02,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:02,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:02,158 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:02,158 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68cfcc2d[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:02,158 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:02,166 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:02,189 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:02,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:02,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:02,190 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:02,190 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:02,190 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:02,190 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,190 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:02,201 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:02,201 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:02,201 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:02,207 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:02,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:02,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:02,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:02,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:02,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:02,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:02,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:02,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:02,228 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:02,228 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:02,228 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:02,228 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:02,228 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:02,230 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:02,230 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:02,230 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:02,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:02,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:02,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:02,230 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:02,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:02,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:02,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:02,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:02,239 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:02,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:02,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:02,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:02,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:02,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:02,256 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,256 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:02,257 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:02,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:02,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:02,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,257 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:02,257 DEBUG [Thread-875] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:02,257 DEBUG [Thread-875] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:02,257 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,257 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:02,257 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:02,257 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:02,258 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:02,258 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:02,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:02,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:02,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:02,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:02,262 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:57:02,312 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230320081f2df6916047549ed96eb56ce68474 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320081f2df6916047549ed96eb56ce68474 2023-03-20 10:57:02,366 DEBUG [Thread-875] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:02,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:02,368 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:02,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:02,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:02,370 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:02,370 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:02,370 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:02,370 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:02,370 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:02,370 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:02,371 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:02,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:02,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:02,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:02,372 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:02,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:02,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:02,372 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,372 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:02,372 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:02,373 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:02,373 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:02,373 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:02,373 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:02,373 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:02,373 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:02,373 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:57:02,373 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. started... 2023-03-20 10:57:02,373 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. as already flushing 2023-03-20 10:57:02,373 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,373 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:02,373 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:57:02,373 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:02,373 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:02,373 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,373 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:02,373 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:02,374 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:02,374 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:02,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:02,375 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:02,375 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:02,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:02,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:02,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:02,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:02,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:02,376 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:02,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3530a8be[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:02,376 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:02,377 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:02,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:02,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:02,377 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:02,377 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:02,377 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:02,378 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,378 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:02,379 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:02,380 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:02,380 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:02,380 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:02,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:02,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:02,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:02,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:02,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:02,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:02,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:02,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:02,386 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:02,386 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:02,386 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:02,386 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:02,386 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:02,395 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:02,395 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:02,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:02,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:02,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:02,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:02,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:02,397 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,398 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:02,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:02,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:02,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:02,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:02,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:02,401 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:02,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:02,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:02,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:02,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,421 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:02,421 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:02,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:02,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:02,421 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:02,421 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,421 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:02,421 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:02,421 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:02,421 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:02,421 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,421 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:02,421 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:02,421 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:02,421 DEBUG [Thread-875] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:02,421 DEBUG [Thread-875] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:02,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:02,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:02,435 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:02,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42958 deadline: 1679309882431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,435 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:02,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42936 deadline: 1679309882432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,521 DEBUG [Thread-875] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:02,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:02,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:02,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:02,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:02,536 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:02,536 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:02,536 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:02,536 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:02,540 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:02,540 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:02,541 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:02,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:02,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:02,541 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:02,542 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,542 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:02,542 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:02,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:02,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:02,542 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:02,545 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:02,545 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:02,545 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:02,545 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:02,545 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:02,549 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:02,549 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:02,549 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:57:02,549 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. started... 2023-03-20 10:57:02,549 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. as already flushing 2023-03-20 10:57:02,549 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:57:02,549 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:02,549 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:02,549 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,549 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:02,549 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:02,554 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:02,554 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:02,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:02,554 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:02,554 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:02,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:02,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:02,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:02,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:02,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:02,556 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:02,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1586b91[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:02,556 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:02,556 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:02,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:02,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:02,556 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:02,556 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:02,556 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:02,557 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,557 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:02,557 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:02,557 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,557 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:02,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,557 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:02,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:02,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:02,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:02,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:02,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:02,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:02,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:02,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:02,559 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:02,559 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:02,559 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:02,559 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:02,559 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:02,559 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:02,560 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:02,560 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:02,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:02,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:02,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:02,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:02,560 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:02,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:02,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:02,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:02,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:02,560 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:02,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:02,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:02,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:02,561 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:02,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:02,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,563 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:02,563 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:02,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:02,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:02,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:02,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:02,572 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,572 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:02,572 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:02,572 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:02,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:02,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:02,573 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,573 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:02,573 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:02,573 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:02,573 DEBUG [Thread-875] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:02,573 DEBUG [Thread-875] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:02,577 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:57:02,673 DEBUG [Thread-875] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:02,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:02,676 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:02,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:02,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:02,677 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:02,677 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:02,677 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:02,677 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:02,677 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:02,677 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:02,678 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:02,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:02,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:02,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:02,679 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:02,679 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,679 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,679 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:02,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:02,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:02,680 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:02,681 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:02,681 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:02,681 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:02,681 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:02,681 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:02,681 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:02,681 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:02,681 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:57:02,681 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. started... 2023-03-20 10:57:02,681 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. as already flushing 2023-03-20 10:57:02,681 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:57:02,681 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:02,681 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:02,681 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,681 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:02,681 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:02,684 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,684 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:02,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:02,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:02,685 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:02,685 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:02,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:02,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:02,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:02,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:02,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:02,688 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:02,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@aa57bf5[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:02,688 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:02,689 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:02,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:02,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:02,689 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:02,689 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:02,689 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:02,691 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,691 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:02,691 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:02,691 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:02,692 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:02,692 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:02,692 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:02,692 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:02,692 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:02,692 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:02,692 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:02,694 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:02,694 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:02,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:02,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:02,694 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:02,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:02,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:02,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:02,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:02,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:02,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:02,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:02,695 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:02,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:02,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:02,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,696 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:02,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:02,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,699 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:02,699 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:02,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:02,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:02,699 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:02,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:02,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:02,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:02,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:02,699 DEBUG [Thread-875] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:02,699 DEBUG [Thread-875] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:02,701 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,701 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:02,701 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:02,701 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,701 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:02,701 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:02,701 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:02,723 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=275, memsize=64.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/A/bf240f17c7314b16bf37781a36bbcc5a 2023-03-20 10:57:02,760 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=64.85 KB at sequenceid=275 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/B/8ea7393145e94257934f33029fef2727 2023-03-20 10:57:02,789 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=64.85 KB at sequenceid=275 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/C/2c79da282a3245bfbc43a19cbeaba00e 2023-03-20 10:57:02,799 DEBUG [Thread-875] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:02,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:02,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:02,802 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/A/bf240f17c7314b16bf37781a36bbcc5a as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/bf240f17c7314b16bf37781a36bbcc5a 2023-03-20 10:57:02,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:02,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:02,805 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:02,805 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:02,805 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:02,805 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:02,805 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:02,805 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:02,807 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:02,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:02,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:02,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:02,808 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,808 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:02,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:02,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:02,809 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:02,809 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,809 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:02,812 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:02,812 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:02,812 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:02,812 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:02,812 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:02,812 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:02,812 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:02,812 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:57:02,812 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/bf240f17c7314b16bf37781a36bbcc5a, entries=200, sequenceid=275, filesize=32.3 K 2023-03-20 10:57:02,812 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. started... 2023-03-20 10:57:02,812 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. as already flushing 2023-03-20 10:57:02,812 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:57:02,812 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:02,813 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:02,813 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,813 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:02,813 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:02,814 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/B/8ea7393145e94257934f33029fef2727 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/8ea7393145e94257934f33029fef2727 2023-03-20 10:57:02,821 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,821 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/8ea7393145e94257934f33029fef2727, entries=150, sequenceid=275, filesize=11.9 K 2023-03-20 10:57:02,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:02,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:02,822 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/C/2c79da282a3245bfbc43a19cbeaba00e as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/2c79da282a3245bfbc43a19cbeaba00e 2023-03-20 10:57:02,824 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:02,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:02,824 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:02,824 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:02,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:02,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:02,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:02,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:02,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c1d7028[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:02,826 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:02,826 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:02,828 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/2c79da282a3245bfbc43a19cbeaba00e, entries=150, sequenceid=275, filesize=11.9 K 2023-03-20 10:57:02,830 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~194.56 KB/199230, heapSize ~498.61 KB/510576, currentSize=6.71 KB/6870 for 9fc4d04a6b2183ce1217c4847d39aded in 1049ms, sequenceid=275, compaction requested=true 2023-03-20 10:57:02,830 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9fc4d04a6b2183ce1217c4847d39aded: 2023-03-20 10:57:02,830 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-03-20 10:57:02,831 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:57:02,831 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-03-20 10:57:02,831 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:57:02,831 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:57:02,834 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 188692 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-03-20 10:57:02,834 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1912): 9fc4d04a6b2183ce1217c4847d39aded/A is initiating minor compaction (all files) 2023-03-20 10:57:02,834 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9fc4d04a6b2183ce1217c4847d39aded/A in TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:57:02,834 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/dc6177d1b44449119f504531a2618b47, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/89fe1a698b064675a497fefe7775f72e, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/9c4d50edf211489fb2e8c8d9ca86e846, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/6e630386549944c0bd784fcc3a6a120a, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/c2dfa925447f4b4286a190f423e5ccf5, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/cb94e2f9fbb94325a40f6784aeb23e94, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/bf240f17c7314b16bf37781a36bbcc5a] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp, totalSize=184.3 K 2023-03-20 10:57:02,834 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 79856 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-03-20 10:57:02,835 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): 9fc4d04a6b2183ce1217c4847d39aded/B is initiating minor compaction (all files) 2023-03-20 10:57:02,835 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9fc4d04a6b2183ce1217c4847d39aded/B in TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:57:02,835 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/ae4c387285774dccaacee853fe0e2362, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/f17638332af0452aa0ff703c7c1828a9, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/e5e7adf94554422f908c7f6fb30dff3a, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/4ea0dfd6f05b42319bf06813fa611e2d, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/4d80978249f54a3696f1e8d053a7603f, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/2ff53b2e1b88479f93749e975bb249a8, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/8ea7393145e94257934f33029fef2727] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp, totalSize=78.0 K 2023-03-20 10:57:02,836 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting dc6177d1b44449119f504531a2618b47, keycount=150, bloomtype=ROW, size=25.5 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1679309815117 2023-03-20 10:57:02,836 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 89fe1a698b064675a497fefe7775f72e, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=161, earliestPutTs=1679309815485 2023-03-20 10:57:02,836 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 9c4d50edf211489fb2e8c8d9ca86e846, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=192, earliestPutTs=1679309816199 2023-03-20 10:57:02,837 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 6e630386549944c0bd784fcc3a6a120a, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=204, earliestPutTs=1679309818364 2023-03-20 10:57:02,837 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting c2dfa925447f4b4286a190f423e5ccf5, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=238, earliestPutTs=1679309818773 2023-03-20 10:57:02,838 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting cb94e2f9fbb94325a40f6784aeb23e94, keycount=100, bloomtype=ROW, size=18.5 K, encoding=NONE, compression=NONE, seqNum=243, earliestPutTs=1679309820972 2023-03-20 10:57:02,838 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting bf240f17c7314b16bf37781a36bbcc5a, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1679309821112 2023-03-20 10:57:02,840 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting ae4c387285774dccaacee853fe0e2362, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1679309815117 2023-03-20 10:57:02,841 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting f17638332af0452aa0ff703c7c1828a9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=161, earliestPutTs=1679309815502 2023-03-20 10:57:02,841 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting e5e7adf94554422f908c7f6fb30dff3a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=192, earliestPutTs=1679309816199 2023-03-20 10:57:02,845 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 4ea0dfd6f05b42319bf06813fa611e2d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=204, earliestPutTs=1679309818364 2023-03-20 10:57:02,846 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 4d80978249f54a3696f1e8d053a7603f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=238, earliestPutTs=1679309818773 2023-03-20 10:57:02,846 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 2ff53b2e1b88479f93749e975bb249a8, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=243, earliestPutTs=1679309820978 2023-03-20 10:57:02,847 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 8ea7393145e94257934f33029fef2727, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1679309821146 2023-03-20 10:57:02,873 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): 9fc4d04a6b2183ce1217c4847d39aded#B#compaction#153 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:57:02,875 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:02,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:02,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:02,875 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:02,876 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:02,876 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,876 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:02,876 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:02,877 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,877 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:02,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:02,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:02,877 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:02,877 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:02,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:02,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:02,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:02,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:02,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:02,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:02,879 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:02,879 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:02,879 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:02,880 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:02,880 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:02,880 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:02,883 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:02,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:02,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:02,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:02,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:02,884 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:02,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:02,885 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:02,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,886 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9fc4d04a6b2183ce1217c4847d39aded#A#compaction#154 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:57:02,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:02,886 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:02,886 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:02,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:02,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:02,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:02,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:02,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:02,888 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,888 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:02,888 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:02,888 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:02,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:02,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:02,888 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:02,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:02,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:02,888 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,888 DEBUG [Thread-875] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:02,888 DEBUG [Thread-875] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:02,888 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:02,889 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:02,889 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:02,889 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:57:02,894 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/B/0d8284b7d09b4719a3fa4f29bf1d4f7c as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/0d8284b7d09b4719a3fa4f29bf1d4f7c 2023-03-20 10:57:02,900 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 9fc4d04a6b2183ce1217c4847d39aded/B of 9fc4d04a6b2183ce1217c4847d39aded into 0d8284b7d09b4719a3fa4f29bf1d4f7c(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:57:02,900 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9fc4d04a6b2183ce1217c4847d39aded: 2023-03-20 10:57:02,900 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded., storeName=9fc4d04a6b2183ce1217c4847d39aded/B, priority=9, startTime=1679309822831; duration=0sec 2023-03-20 10:57:02,900 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:57:02,900 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-03-20 10:57:02,905 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 79856 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-03-20 10:57:02,905 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): 9fc4d04a6b2183ce1217c4847d39aded/C is initiating minor compaction (all files) 2023-03-20 10:57:02,905 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9fc4d04a6b2183ce1217c4847d39aded/C in TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:57:02,906 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/7725125e02a840699e78029a30adff1d, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/329cfc79a9a84b808f795f9f880cdded, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/9bdb3eb481744df19b69cf78e641b60e, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/b501de08c8b141b588706b7dfa423be2, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/99751ba587ef43dc8e1d7e873a1b721a, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/dd58f713e4954992a5600d4b2615b513, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/2c79da282a3245bfbc43a19cbeaba00e] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp, totalSize=78.0 K 2023-03-20 10:57:02,910 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 7725125e02a840699e78029a30adff1d, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1679309815117 2023-03-20 10:57:02,911 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 329cfc79a9a84b808f795f9f880cdded, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=161, earliestPutTs=1679309815502 2023-03-20 10:57:02,911 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 9bdb3eb481744df19b69cf78e641b60e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=192, earliestPutTs=1679309816199 2023-03-20 10:57:02,913 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting b501de08c8b141b588706b7dfa423be2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=204, earliestPutTs=1679309818364 2023-03-20 10:57:02,914 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 99751ba587ef43dc8e1d7e873a1b721a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=238, earliestPutTs=1679309818773 2023-03-20 10:57:02,915 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting dd58f713e4954992a5600d4b2615b513, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=243, earliestPutTs=1679309820978 2023-03-20 10:57:02,916 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 2c79da282a3245bfbc43a19cbeaba00e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1679309821146 2023-03-20 10:57:02,946 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): 9fc4d04a6b2183ce1217c4847d39aded#C#compaction#155 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:57:02,976 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/C/da5f6e9d88c54dd786d187d77b8c6b50 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/da5f6e9d88c54dd786d187d77b8c6b50 2023-03-20 10:57:02,989 DEBUG [Thread-875] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:02,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:02,990 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 9fc4d04a6b2183ce1217c4847d39aded/C of 9fc4d04a6b2183ce1217c4847d39aded into da5f6e9d88c54dd786d187d77b8c6b50(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:57:02,991 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9fc4d04a6b2183ce1217c4847d39aded: 2023-03-20 10:57:02,991 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded., storeName=9fc4d04a6b2183ce1217c4847d39aded/C, priority=9, startTime=1679309822831; duration=0sec 2023-03-20 10:57:02,991 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:57:02,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:02,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:02,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:02,993 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:02,993 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:02,994 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:02,994 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:02,994 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:02,994 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:02,995 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:02,995 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,996 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:02,996 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:02,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:02,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:02,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:02,997 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:02,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:02,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:02,998 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:02,998 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:02,998 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:02,998 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:02,998 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:02,999 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:02,999 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:02,999 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:02,999 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:57:02,999 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. started... 2023-03-20 10:57:02,999 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9fc4d04a6b2183ce1217c4847d39aded 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-20 10:57:03,000 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9fc4d04a6b2183ce1217c4847d39aded, store=A 2023-03-20 10:57:03,000 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:03,001 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9fc4d04a6b2183ce1217c4847d39aded, store=B 2023-03-20 10:57:03,001 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:03,001 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9fc4d04a6b2183ce1217c4847d39aded, store=C 2023-03-20 10:57:03,001 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:03,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 9fc4d04a6b2183ce1217c4847d39aded 2023-03-20 10:57:03,030 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. as already flushing 2023-03-20 10:57:03,054 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230320994ab891137049a6bb9dc1929b24b518 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320994ab891137049a6bb9dc1929b24b518 2023-03-20 10:57:03,076 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=285, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/A/70e74d1d86e4431fa0adc701a4a9a589 2023-03-20 10:57:03,107 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=285 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/B/7f987e294b904f2196e5e9b95a76c5e8 2023-03-20 10:57:03,134 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=285 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/C/597339b3b1d842eaa8d467c6d47ece72 2023-03-20 10:57:03,141 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/A/70e74d1d86e4431fa0adc701a4a9a589 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/70e74d1d86e4431fa0adc701a4a9a589 2023-03-20 10:57:03,152 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/70e74d1d86e4431fa0adc701a4a9a589, entries=250, sequenceid=285, filesize=39.2 K 2023-03-20 10:57:03,154 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=759, cellSize=34755, totalHeapSize=79176, min timestamp=1679309823045, max timestamp=1679309823151Num uniques 859; 2023-03-20 10:57:03,154 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=34755, suffix heap size=117064,new segment heap size=79176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=759 2023-03-20 10:57:03,156 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=759, cellSize=34755, totalHeapSize=79176, min timestamp=1679309823045, max timestamp=1679309823151Num uniques 859; 2023-03-20 10:57:03,156 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=34755, suffix heap size=117064,new segment heap size=79176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=759 2023-03-20 10:57:03,157 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/B/7f987e294b904f2196e5e9b95a76c5e8 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/7f987e294b904f2196e5e9b95a76c5e8 2023-03-20 10:57:03,160 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=759, cellSize=34755, totalHeapSize=79176, min timestamp=1679309823045, max timestamp=1679309823151Num uniques 859; 2023-03-20 10:57:03,160 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=34755, suffix heap size=117064,new segment heap size=79176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=759 2023-03-20 10:57:03,172 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/7f987e294b904f2196e5e9b95a76c5e8, entries=100, sequenceid=285, filesize=9.5 K 2023-03-20 10:57:03,174 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/C/597339b3b1d842eaa8d467c6d47ece72 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/597339b3b1d842eaa8d467c6d47ece72 2023-03-20 10:57:03,184 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/597339b3b1d842eaa8d467c6d47ece72, entries=100, sequenceid=285, filesize=9.5 K 2023-03-20 10:57:03,185 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=147.60 KB/151140 for 9fc4d04a6b2183ce1217c4847d39aded in 186ms, sequenceid=285, compaction requested=false 2023-03-20 10:57:03,186 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9fc4d04a6b2183ce1217c4847d39aded: 2023-03-20 10:57:03,186 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:57:03,186 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:03,186 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:03,186 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,186 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:03,186 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:03,192 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9fc4d04a6b2183ce1217c4847d39aded 3/3 column families, dataSize=154.31 KB heapSize=388.50 KB 2023-03-20 10:57:03,192 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9fc4d04a6b2183ce1217c4847d39aded, store=A 2023-03-20 10:57:03,192 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:57:03,192 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9fc4d04a6b2183ce1217c4847d39aded, store=B 2023-03-20 10:57:03,192 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:57:03,192 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9fc4d04a6b2183ce1217c4847d39aded, store=C 2023-03-20 10:57:03,192 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:57:03,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 9fc4d04a6b2183ce1217c4847d39aded 2023-03-20 10:57:03,202 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:03,206 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,206 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:03,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:03,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:03,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:03,206 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:03,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:03,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:03,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:03,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:03,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2cd85e4d[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:03,215 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:03,216 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:03,217 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:03,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:03,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:03,218 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:03,218 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:03,218 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:03,219 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:03,219 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:03,219 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:03,220 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:03,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:03,220 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,220 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:03,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:03,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:03,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:03,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:03,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:03,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:03,223 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:03,223 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:03,223 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:03,223 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:03,223 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:03,223 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:03,228 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:03,228 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:03,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:03,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:03,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:03,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:03,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:03,228 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:03,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:03,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:03,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:03,234 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:03,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42958 deadline: 1679309883220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,235 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:03,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42936 deadline: 1679309883234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:03,236 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:03,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:03,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:03,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,240 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:03,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:03,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,242 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,242 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:03,242 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:03,242 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,242 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:03,242 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:03,242 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:03,242 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:03,242 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:03,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:03,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:03,242 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:03,242 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:57:03,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:03,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:03,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:03,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:03,243 DEBUG [Thread-875] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:03,243 DEBUG [Thread-875] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:03,259 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230320057d4e48e2c74bd18d7b92cd9aaf0320 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320057d4e48e2c74bd18d7b92cd9aaf0320 2023-03-20 10:57:03,280 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:03,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:42986 deadline: 1679309883277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,283 DEBUG [Thread-867] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8175 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded., hostname=jenkins-hbase20.apache.org,36041,1679309777062, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-20 10:57:03,291 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=313, memsize=51.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/A/b678fc7564704a5a8771a284bcfea8eb 2023-03-20 10:57:03,327 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=313 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/B/c8b3401dc8db479086174beec11ed34b 2023-03-20 10:57:03,337 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:03,337 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:03,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42958 deadline: 1679309883335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42936 deadline: 1679309883336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,343 DEBUG [Thread-875] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:03,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:03,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:03,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:03,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:03,374 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:03,374 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:03,374 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:03,374 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:03,375 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:03,375 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:03,378 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:03,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:03,378 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:03,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:03,378 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,378 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:03,379 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:03,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:03,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:03,379 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:03,380 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/A/d0ce449621844463a5b8566e859a72da as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/d0ce449621844463a5b8566e859a72da 2023-03-20 10:57:03,391 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 9fc4d04a6b2183ce1217c4847d39aded/A of 9fc4d04a6b2183ce1217c4847d39aded into d0ce449621844463a5b8566e859a72da(size=25.9 K), total size for store is 65.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:57:03,391 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9fc4d04a6b2183ce1217c4847d39aded: 2023-03-20 10:57:03,391 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded., storeName=9fc4d04a6b2183ce1217c4847d39aded/A, priority=9, startTime=1679309822830; duration=0sec 2023-03-20 10:57:03,391 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:57:03,393 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:03,393 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:03,393 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:03,393 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:03,393 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:03,393 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=313 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/C/7a39d3db07104287909c7e6aac06acf9 2023-03-20 10:57:03,402 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/A/b678fc7564704a5a8771a284bcfea8eb as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/b678fc7564704a5a8771a284bcfea8eb 2023-03-20 10:57:03,406 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:03,406 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:03,406 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:57:03,406 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. started... 2023-03-20 10:57:03,406 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. as already flushing 2023-03-20 10:57:03,406 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:57:03,406 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:03,406 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:03,406 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,406 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:03,406 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:03,411 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:03,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:03,412 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:03,412 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:03,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:03,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:03,413 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:03,413 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/b678fc7564704a5a8771a284bcfea8eb, entries=300, sequenceid=313, filesize=46.1 K 2023-03-20 10:57:03,416 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/B/c8b3401dc8db479086174beec11ed34b as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/c8b3401dc8db479086174beec11ed34b 2023-03-20 10:57:03,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:03,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:03,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:03,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e9eaca4[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:03,423 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:03,423 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:03,425 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:03,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:03,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:03,425 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:03,425 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:03,425 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:03,425 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,425 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:03,427 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,427 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:03,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:03,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:03,427 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:03,427 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:03,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:03,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:03,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:03,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:03,431 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/c8b3401dc8db479086174beec11ed34b, entries=150, sequenceid=313, filesize=11.9 K 2023-03-20 10:57:03,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:03,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,433 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/C/7a39d3db07104287909c7e6aac06acf9 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/7a39d3db07104287909c7e6aac06acf9 2023-03-20 10:57:03,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:03,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:03,433 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:03,433 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:03,433 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:03,433 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:03,433 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:03,437 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:03,437 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:03,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:03,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:03,438 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,437 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:03,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:03,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:03,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:03,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:03,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:03,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:03,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:03,440 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,440 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/7a39d3db07104287909c7e6aac06acf9, entries=150, sequenceid=313, filesize=11.9 K 2023-03-20 10:57:03,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:03,441 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~388.45 KB/397776, currentSize=46.96 KB/48090 for 9fc4d04a6b2183ce1217c4847d39aded in 249ms, sequenceid=313, compaction requested=true 2023-03-20 10:57:03,441 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9fc4d04a6b2183ce1217c4847d39aded: 2023-03-20 10:57:03,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:03,441 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 10:57:03,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:03,442 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:57:03,442 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 10:57:03,442 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:57:03,443 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:57:03,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,444 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 113820 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 10:57:03,444 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1912): 9fc4d04a6b2183ce1217c4847d39aded/A is initiating minor compaction (all files) 2023-03-20 10:57:03,444 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9fc4d04a6b2183ce1217c4847d39aded/A in TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:57:03,444 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/d0ce449621844463a5b8566e859a72da, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/70e74d1d86e4431fa0adc701a4a9a589, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/b678fc7564704a5a8771a284bcfea8eb] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp, totalSize=111.2 K 2023-03-20 10:57:03,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:03,445 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting d0ce449621844463a5b8566e859a72da, keycount=150, bloomtype=ROW, size=25.9 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1679309821146 2023-03-20 10:57:03,446 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34592 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 10:57:03,446 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:03,447 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): 9fc4d04a6b2183ce1217c4847d39aded/B is initiating minor compaction (all files) 2023-03-20 10:57:03,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:03,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:03,447 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 70e74d1d86e4431fa0adc701a4a9a589, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1679309821788 2023-03-20 10:57:03,446 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:03,447 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:03,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:03,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:03,446 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,447 DEBUG [Thread-875] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:03,447 DEBUG [Thread-875] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:03,447 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:57:03,447 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:03,448 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:03,448 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,448 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:03,448 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:03,448 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:03,447 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9fc4d04a6b2183ce1217c4847d39aded/B in TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:57:03,448 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting b678fc7564704a5a8771a284bcfea8eb, keycount=300, bloomtype=ROW, size=46.1 K, encoding=NONE, compression=NONE, seqNum=313, earliestPutTs=1679309823128 2023-03-20 10:57:03,448 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/0d8284b7d09b4719a3fa4f29bf1d4f7c, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/7f987e294b904f2196e5e9b95a76c5e8, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/c8b3401dc8db479086174beec11ed34b] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp, totalSize=33.8 K 2023-03-20 10:57:03,448 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 0d8284b7d09b4719a3fa4f29bf1d4f7c, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1679309821146 2023-03-20 10:57:03,449 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 7f987e294b904f2196e5e9b95a76c5e8, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1679309822994 2023-03-20 10:57:03,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:03,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:03,454 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting c8b3401dc8db479086174beec11ed34b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=313, earliestPutTs=1679309823181 2023-03-20 10:57:03,469 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): 9fc4d04a6b2183ce1217c4847d39aded#B#compaction#163 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:57:03,481 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9fc4d04a6b2183ce1217c4847d39aded#A#compaction#162 average throughput is 1.52 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:57:03,508 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/B/e327e36297f34a908a080974c7c58f8b as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/e327e36297f34a908a080974c7c58f8b 2023-03-20 10:57:03,515 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9fc4d04a6b2183ce1217c4847d39aded/B of 9fc4d04a6b2183ce1217c4847d39aded into e327e36297f34a908a080974c7c58f8b(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:57:03,515 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9fc4d04a6b2183ce1217c4847d39aded: 2023-03-20 10:57:03,515 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded., storeName=9fc4d04a6b2183ce1217c4847d39aded/B, priority=13, startTime=1679309823442; duration=0sec 2023-03-20 10:57:03,515 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:57:03,515 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 10:57:03,522 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34592 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 10:57:03,523 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): 9fc4d04a6b2183ce1217c4847d39aded/C is initiating minor compaction (all files) 2023-03-20 10:57:03,523 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9fc4d04a6b2183ce1217c4847d39aded/C in TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:57:03,523 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/da5f6e9d88c54dd786d187d77b8c6b50, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/597339b3b1d842eaa8d467c6d47ece72, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/7a39d3db07104287909c7e6aac06acf9] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp, totalSize=33.8 K 2023-03-20 10:57:03,524 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting da5f6e9d88c54dd786d187d77b8c6b50, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1679309821146 2023-03-20 10:57:03,524 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 597339b3b1d842eaa8d467c6d47ece72, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1679309822994 2023-03-20 10:57:03,525 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 7a39d3db07104287909c7e6aac06acf9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=313, earliestPutTs=1679309823181 2023-03-20 10:57:03,539 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): 9fc4d04a6b2183ce1217c4847d39aded#C#compaction#164 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:57:03,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 9fc4d04a6b2183ce1217c4847d39aded 2023-03-20 10:57:03,540 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9fc4d04a6b2183ce1217c4847d39aded 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-20 10:57:03,542 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9fc4d04a6b2183ce1217c4847d39aded, store=A 2023-03-20 10:57:03,542 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:03,542 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9fc4d04a6b2183ce1217c4847d39aded, store=B 2023-03-20 10:57:03,542 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:03,542 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9fc4d04a6b2183ce1217c4847d39aded, store=C 2023-03-20 10:57:03,542 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:03,548 DEBUG [Thread-875] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:03,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:03,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:03,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:03,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:03,553 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:03,553 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:03,553 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:03,553 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:03,554 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:03,554 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:03,555 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,555 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,555 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:03,560 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:03,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:03,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:03,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:03,561 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:03,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:03,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:03,562 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:03,567 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:03,567 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:03,567 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:03,567 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:03,567 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:03,568 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:03,568 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:03,568 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:57:03,569 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. started... 2023-03-20 10:57:03,572 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032058be4299b00e4f738fc6be540f737146 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032058be4299b00e4f738fc6be540f737146 2023-03-20 10:57:03,579 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. as already flushing 2023-03-20 10:57:03,579 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:57:03,579 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:03,579 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:03,579 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,579 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:03,579 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:03,569 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/C/89f232e1b943419cbf08edf2229a6095 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/89f232e1b943419cbf08edf2229a6095 2023-03-20 10:57:03,581 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:03,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:03,581 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:03,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:03,582 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:03,582 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:03,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:03,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:03,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:03,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:03,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fc51851[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:03,584 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:03,584 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:03,585 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:03,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:03,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:03,585 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:03,585 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:03,585 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:03,585 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,586 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:03,587 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:03,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42936 deadline: 1679309883581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,588 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9fc4d04a6b2183ce1217c4847d39aded/C of 9fc4d04a6b2183ce1217c4847d39aded into 89f232e1b943419cbf08edf2229a6095(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:57:03,588 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9fc4d04a6b2183ce1217c4847d39aded: 2023-03-20 10:57:03,588 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded., storeName=9fc4d04a6b2183ce1217c4847d39aded/C, priority=13, startTime=1679309823443; duration=0sec 2023-03-20 10:57:03,588 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:57:03,588 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:03,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42958 deadline: 1679309883584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,589 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:03,589 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:03,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:03,589 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:03,589 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:03,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:03,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:03,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:03,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:03,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:03,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:03,595 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:03,595 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:03,595 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:03,595 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:03,595 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:03,595 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:03,597 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:03,597 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:03,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:03,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:03,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:03,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:03,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:03,597 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:03,597 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:03,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:03,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:03,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:03,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:03,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:03,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:03,599 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:03,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:03,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,603 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:03,603 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:03,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:03,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:03,604 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:03,604 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,604 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:03,604 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:03,604 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,604 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:03,604 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:03,604 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:03,604 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:03,604 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:03,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:03,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:03,604 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:57:03,604 DEBUG [Thread-875] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:03,604 DEBUG [Thread-875] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:03,622 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=327, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/A/c9d269d945524ba2a12f5fa5c4b79d8c 2023-03-20 10:57:03,691 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:03,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42936 deadline: 1679309883688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,692 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:03,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42958 deadline: 1679309883690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,704 DEBUG [Thread-875] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:03,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:03,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:03,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:03,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:03,713 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:03,713 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:03,713 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:03,713 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:03,714 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:03,714 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:03,718 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:03,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:03,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:03,718 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:03,719 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,719 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:03,720 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:03,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:03,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:03,721 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:03,721 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:03,721 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:03,721 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:03,721 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:03,722 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:03,722 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:03,722 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:03,722 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:57:03,722 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. started... 2023-03-20 10:57:03,722 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. as already flushing 2023-03-20 10:57:03,722 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:57:03,722 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:03,722 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:03,722 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,722 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:03,722 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:03,738 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:03,739 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:03,739 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:03,742 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:03,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:03,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:03,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:03,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:03,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:03,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:03,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d0acb2a[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:03,751 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:03,751 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:03,752 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:03,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:03,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:03,752 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:03,752 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:03,752 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,752 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:03,752 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:03,753 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:03,753 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:03,753 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:03,754 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:03,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:03,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:03,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:03,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:03,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:03,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:03,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:03,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:03,759 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:03,759 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:03,759 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:03,759 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:03,759 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:03,762 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:03,762 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:03,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:03,762 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:03,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:03,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:03,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:03,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:03,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:03,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:03,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:03,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:03,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:03,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:03,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:03,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:03,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:03,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,770 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,771 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,782 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:03,782 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:03,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:03,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:03,784 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,784 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:03,784 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:03,784 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,784 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:03,784 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:03,784 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:03,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:03,784 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:03,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:03,784 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:03,784 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:03,784 DEBUG [Thread-875] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:03,785 DEBUG [Thread-875] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:03,885 DEBUG [Thread-875] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:03,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:03,899 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:03,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42936 deadline: 1679309883899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:03,902 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:03,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42958 deadline: 1679309883901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:03,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:03,914 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:03,914 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:03,914 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:03,914 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:03,915 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:03,915 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:03,918 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:03,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:03,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:03,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:03,919 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:03,919 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:03,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:03,919 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:03,920 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,920 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:03,920 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:03,920 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:03,921 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:03,921 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:03,921 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:03,921 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:03,921 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:03,921 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:57:03,921 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. started... 2023-03-20 10:57:03,921 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. as already flushing 2023-03-20 10:57:03,921 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:57:03,921 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:03,921 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:03,921 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,921 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:03,921 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:03,930 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:03,930 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:03,930 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:03,937 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/A/9391e6713cce47cd8ddff9b6c3f39905 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/9391e6713cce47cd8ddff9b6c3f39905 2023-03-20 10:57:03,938 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:03,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:03,945 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9fc4d04a6b2183ce1217c4847d39aded/A of 9fc4d04a6b2183ce1217c4847d39aded into 9391e6713cce47cd8ddff9b6c3f39905(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:57:03,945 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9fc4d04a6b2183ce1217c4847d39aded: 2023-03-20 10:57:03,945 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded., storeName=9fc4d04a6b2183ce1217c4847d39aded/A, priority=13, startTime=1679309823441; duration=0sec 2023-03-20 10:57:03,945 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:57:03,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:03,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:03,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:03,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:03,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:03,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2aea554a[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:03,967 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:03,967 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:03,974 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:03,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:03,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:03,974 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,974 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:03,978 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:03,978 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:03,978 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:03,987 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:03,987 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:03,987 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:03,987 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:03,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:03,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:03,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:03,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:03,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:03,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:03,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:03,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:03,991 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:03,991 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:03,991 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:03,991 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:03,991 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:03,992 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:03,992 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:03,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:03,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:03,992 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:03,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:03,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:03,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:03,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:03,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:03,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:03,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:03,993 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:03,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:03,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:03,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:03,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:04,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:04,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:04,000 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:04,013 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:04,013 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:04,013 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:04,013 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:04,013 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:04,014 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:04,014 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:04,014 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:04,014 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:04,014 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:04,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:04,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:04,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:04,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:04,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:04,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:04,019 DEBUG [Thread-875] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:04,019 DEBUG [Thread-875] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:04,062 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=327 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/B/f8e15d6f47c0408bac3066fa053a54fc 2023-03-20 10:57:04,075 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=327 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/C/6fdd90dc469941cbb126702523ea3a18 2023-03-20 10:57:04,080 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/A/c9d269d945524ba2a12f5fa5c4b79d8c as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/c9d269d945524ba2a12f5fa5c4b79d8c 2023-03-20 10:57:04,087 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/c9d269d945524ba2a12f5fa5c4b79d8c, entries=150, sequenceid=327, filesize=25.4 K 2023-03-20 10:57:04,088 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/B/f8e15d6f47c0408bac3066fa053a54fc as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/f8e15d6f47c0408bac3066fa053a54fc 2023-03-20 10:57:04,099 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/f8e15d6f47c0408bac3066fa053a54fc, entries=150, sequenceid=327, filesize=11.9 K 2023-03-20 10:57:04,103 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/C/6fdd90dc469941cbb126702523ea3a18 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/6fdd90dc469941cbb126702523ea3a18 2023-03-20 10:57:04,119 DEBUG [Thread-875] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:04,121 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/6fdd90dc469941cbb126702523ea3a18, entries=150, sequenceid=327, filesize=11.9 K 2023-03-20 10:57:04,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:04,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:04,125 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 9fc4d04a6b2183ce1217c4847d39aded in 585ms, sequenceid=327, compaction requested=false 2023-03-20 10:57:04,125 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9fc4d04a6b2183ce1217c4847d39aded: 2023-03-20 10:57:04,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:04,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:04,138 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:04,138 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:04,138 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:04,138 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:04,141 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:04,141 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:04,142 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:04,143 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:04,143 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:04,144 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:04,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:04,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:04,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:04,145 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:04,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:04,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:04,146 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:04,149 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:04,150 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:04,150 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:04,150 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:04,150 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:04,154 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:04,154 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:04,158 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:57:04,158 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. started... 2023-03-20 10:57:04,158 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9fc4d04a6b2183ce1217c4847d39aded 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-20 10:57:04,158 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9fc4d04a6b2183ce1217c4847d39aded, store=A 2023-03-20 10:57:04,159 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:04,159 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9fc4d04a6b2183ce1217c4847d39aded, store=B 2023-03-20 10:57:04,159 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:04,159 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9fc4d04a6b2183ce1217c4847d39aded, store=C 2023-03-20 10:57:04,159 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:04,222 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. as already flushing 2023-03-20 10:57:04,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 9fc4d04a6b2183ce1217c4847d39aded 2023-03-20 10:57:04,261 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:04,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42958 deadline: 1679309884256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:04,262 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:04,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42936 deadline: 1679309884256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:04,367 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:04,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42958 deadline: 1679309884363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:04,371 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:04,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42936 deadline: 1679309884370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:04,573 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:04,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42958 deadline: 1679309884571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:04,577 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:04,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42936 deadline: 1679309884574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:04,661 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032069ddaf8d033c49389d26435eceb35940 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032069ddaf8d033c49389d26435eceb35940 2023-03-20 10:57:04,689 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=349, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/A/8eecf89b87674f768f83df4fa1a9fec5 2023-03-20 10:57:04,762 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=349 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/B/00fd09649c1344c09dff2e52e338d56a 2023-03-20 10:57:04,795 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=349 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/C/e40adb40dbb04e68b5c84c789ec66876 2023-03-20 10:57:04,807 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/A/8eecf89b87674f768f83df4fa1a9fec5 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/8eecf89b87674f768f83df4fa1a9fec5 2023-03-20 10:57:04,813 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/8eecf89b87674f768f83df4fa1a9fec5, entries=150, sequenceid=349, filesize=25.4 K 2023-03-20 10:57:04,815 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/B/00fd09649c1344c09dff2e52e338d56a as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/00fd09649c1344c09dff2e52e338d56a 2023-03-20 10:57:04,824 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/00fd09649c1344c09dff2e52e338d56a, entries=150, sequenceid=349, filesize=11.9 K 2023-03-20 10:57:04,826 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/C/e40adb40dbb04e68b5c84c789ec66876 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/e40adb40dbb04e68b5c84c789ec66876 2023-03-20 10:57:04,843 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/e40adb40dbb04e68b5c84c789ec66876, entries=150, sequenceid=349, filesize=11.9 K 2023-03-20 10:57:04,844 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 9fc4d04a6b2183ce1217c4847d39aded in 686ms, sequenceid=349, compaction requested=true 2023-03-20 10:57:04,845 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9fc4d04a6b2183ce1217c4847d39aded: 2023-03-20 10:57:04,845 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:57:04,845 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:04,845 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:04,845 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:04,845 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:04,845 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:04,848 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:04,849 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:04,849 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:04,850 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:04,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:04,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:04,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:04,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:04,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:04,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:04,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:04,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:04,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:04,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6764a683[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:04,853 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:04,853 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:04,863 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:04,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:04,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:04,863 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:04,863 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:04,863 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:04,863 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:04,863 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:04,864 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:04,864 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:04,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:04,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:04,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:04,864 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:04,864 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:04,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:04,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:04,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:04,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:04,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:04,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:04,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:04,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:04,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:04,866 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:04,866 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:04,866 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:04,866 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:04,866 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:04,867 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:04,867 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:04,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:04,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:04,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:04,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:04,867 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:04,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:04,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:04,868 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:04,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:04,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:04,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:04,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:04,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:04,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:04,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:04,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:04,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:04,872 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:04,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:04,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 9fc4d04a6b2183ce1217c4847d39aded 2023-03-20 10:57:04,879 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9fc4d04a6b2183ce1217c4847d39aded 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-20 10:57:04,879 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9fc4d04a6b2183ce1217c4847d39aded, store=A 2023-03-20 10:57:04,880 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:04,880 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9fc4d04a6b2183ce1217c4847d39aded, store=B 2023-03-20 10:57:04,880 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:04,880 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9fc4d04a6b2183ce1217c4847d39aded, store=C 2023-03-20 10:57:04,880 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:04,881 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:04,881 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:04,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:04,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:04,881 DEBUG [Thread-875] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:04,882 DEBUG [Thread-875] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:04,881 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:04,882 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:04,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:04,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:04,881 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:04,883 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:04,883 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:04,883 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:04,883 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:04,881 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:04,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:04,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:04,929 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:04,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42958 deadline: 1679309884926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:04,930 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:04,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42936 deadline: 1679309884926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:04,931 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303201ebb2a4b260846c8b4fffb23a0fe7dd9 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303201ebb2a4b260846c8b4fffb23a0fe7dd9 2023-03-20 10:57:04,970 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=362, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/A/48549e470b2c4c119765fca40fbff98c 2023-03-20 10:57:04,982 DEBUG [Thread-875] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:04,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:04,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:04,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:04,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:04,989 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:04,989 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:04,989 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:04,989 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:04,989 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:04,989 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:04,996 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:04,996 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:04,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:04,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:04,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:04,997 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:04,997 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:04,997 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:04,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:04,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:04,997 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:04,998 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:04,998 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:04,998 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:04,998 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:04,998 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:05,000 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:05,001 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:05,010 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:57:05,010 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. started... 2023-03-20 10:57:05,010 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. as already flushing 2023-03-20 10:57:05,010 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:57:05,010 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:05,010 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:05,010 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,010 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:05,010 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:05,012 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:05,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:05,014 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:05,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:05,018 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:05,018 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:05,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:05,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:05,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:05,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:05,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@643ba8fe[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:05,020 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:05,020 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:05,021 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:05,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:05,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:05,021 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:05,021 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,021 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:05,021 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:05,021 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:05,022 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,022 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:05,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:05,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:05,022 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:05,022 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:05,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:05,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:05,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:05,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:05,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:05,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:05,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:05,026 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:05,026 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:05,026 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:05,026 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:05,026 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:05,027 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:05,027 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:05,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:05,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:05,027 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:05,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:05,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:05,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:05,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:05,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:05,028 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:05,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:05,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:05,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:05,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:05,030 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:05,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:05,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,032 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:05,032 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:05,032 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:05,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:05,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:05,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:05,032 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:05,033 DEBUG [Thread-875] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:05,033 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:57:05,032 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:05,033 DEBUG [Thread-875] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:05,033 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:05,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:05,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:05,033 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,033 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:05,033 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:05,033 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:05,033 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:05,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42958 deadline: 1679309885031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,034 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:05,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42936 deadline: 1679309885031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,133 DEBUG [Thread-875] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:05,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:05,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:05,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:05,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:05,135 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:05,135 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:05,135 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:05,135 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:05,136 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:05,136 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:05,136 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,136 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:05,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:05,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:05,137 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,137 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:05,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:05,145 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:05,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:05,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:05,146 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:05,146 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:05,146 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:05,146 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:05,146 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:05,146 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:05,149 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:05,149 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:57:05,149 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:05,149 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. started... 2023-03-20 10:57:05,150 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. as already flushing 2023-03-20 10:57:05,150 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:57:05,150 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:05,150 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:05,150 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,150 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:05,150 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:05,151 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,151 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:05,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:05,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:05,151 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:05,151 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:05,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:05,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:05,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:05,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:05,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:05,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f737cc8[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:05,152 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:05,152 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:05,153 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:05,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:05,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:05,153 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:05,153 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,153 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:05,153 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:05,153 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:05,153 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:05,153 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,153 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:05,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:05,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:05,154 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:05,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:05,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:05,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:05,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:05,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:05,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:05,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:05,155 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:05,155 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:05,155 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:05,155 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:05,155 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:05,156 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:05,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:05,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:05,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:05,156 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:05,156 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:05,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:05,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:05,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:05,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:05,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:05,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:05,157 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:05,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:05,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:05,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,159 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:05,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:05,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,161 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,161 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:05,161 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:05,161 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:05,161 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:05,161 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:57:05,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:05,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:05,161 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:05,161 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:05,161 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:05,161 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,161 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:05,161 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:05,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:05,161 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:05,161 DEBUG [Thread-875] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:05,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:05,161 DEBUG [Thread-875] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:05,236 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:05,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42936 deadline: 1679309885235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,237 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:05,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42958 deadline: 1679309885235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,261 DEBUG [Thread-875] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:05,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:05,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:05,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:05,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:05,264 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:05,264 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:05,264 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:05,264 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:05,264 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:05,264 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:05,265 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,265 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:05,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:05,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:05,265 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,265 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:05,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:05,266 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:05,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:05,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:05,266 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:05,267 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:05,267 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:05,267 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:05,267 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:05,267 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:05,268 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:05,268 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:05,268 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:57:05,268 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. started... 2023-03-20 10:57:05,268 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. as already flushing 2023-03-20 10:57:05,268 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:57:05,268 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:05,268 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:05,268 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,268 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:05,268 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:05,269 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,269 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:05,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:05,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:05,269 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:05,269 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:05,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:05,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:05,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:05,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:05,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:05,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7cb2669[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:05,271 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:05,271 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:05,272 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:05,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:05,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:05,273 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:05,273 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:05,273 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:05,273 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,274 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:05,274 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:05,274 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:05,274 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:05,274 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:05,274 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:05,274 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:05,274 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:05,274 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:05,274 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:05,275 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:05,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:05,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:05,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:05,276 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:05,276 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:05,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:05,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:05,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:05,279 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:05,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:05,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:05,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:05,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:05,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:05,280 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:05,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:05,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,287 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,287 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:05,287 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:05,287 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:05,288 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:05,287 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,288 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:05,288 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:05,288 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:05,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:05,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:05,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:05,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:05,288 DEBUG [Thread-875] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:05,288 DEBUG [Thread-875] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:05,288 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:57:05,288 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:05,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:05,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:05,385 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=362 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/B/017afb26562c4e0894f44f1ab1bbc8fb 2023-03-20 10:57:05,388 DEBUG [Thread-875] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:05,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:05,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:05,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:05,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:05,392 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:05,392 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:05,392 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:05,392 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:05,400 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:05,400 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:05,401 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:05,401 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:05,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:05,402 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,402 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:05,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:05,402 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:05,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:05,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:05,403 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:05,403 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:05,403 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:05,403 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:05,403 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:05,403 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:05,403 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:05,403 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:05,403 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:57:05,404 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. started... 2023-03-20 10:57:05,404 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. as already flushing 2023-03-20 10:57:05,404 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:57:05,404 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:05,404 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:05,404 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,404 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:05,404 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:05,405 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,405 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:05,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:05,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:05,406 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:05,406 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:05,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:05,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:05,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:05,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:05,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:05,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4559c8ad[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:05,407 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:05,407 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:05,407 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:05,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:05,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:05,408 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:05,408 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:05,408 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,408 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:05,408 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:05,409 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:05,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:05,409 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:05,409 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:05,409 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:05,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:05,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:05,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:05,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:05,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:05,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:05,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:05,411 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:05,411 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:05,411 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:05,411 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:05,411 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:05,412 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:05,412 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:05,412 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:05,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:05,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:05,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:05,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:05,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:05,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:05,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:05,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:05,413 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:05,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:05,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:05,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:05,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,414 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:05,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:05,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,416 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:05,416 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:05,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:05,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:05,417 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,417 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:05,417 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:05,417 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,417 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:05,417 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:05,417 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:05,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:05,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:05,418 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:05,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:05,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:05,419 DEBUG [Thread-875] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:05,419 DEBUG [Thread-875] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:05,519 DEBUG [Thread-875] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:05,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:05,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:05,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:05,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:05,534 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:05,534 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:05,534 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:05,534 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:05,534 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:05,534 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:05,535 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,535 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:05,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:05,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:05,535 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,535 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:05,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:05,537 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:05,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:05,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:05,537 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:05,544 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:05,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42936 deadline: 1679309885538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,546 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:05,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42958 deadline: 1679309885539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,550 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:05,551 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:05,551 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:05,551 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:05,551 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:05,551 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:05,551 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:05,551 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:57:05,551 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. started... 2023-03-20 10:57:05,551 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. as already flushing 2023-03-20 10:57:05,552 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:57:05,552 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:05,552 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:05,552 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,552 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:05,552 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:05,553 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,553 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:05,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:05,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:05,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:05,553 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:05,553 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:05,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:05,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:05,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:05,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:05,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c1cf4a0[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:05,555 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:05,555 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:05,556 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:05,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:05,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:05,556 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:05,556 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:05,556 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:05,556 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,556 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:05,557 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,557 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:05,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:05,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:05,557 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:05,557 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:05,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:05,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:05,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:05,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:05,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:05,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:05,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:05,560 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:05,560 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:05,560 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:05,560 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:05,561 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:05,562 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:05,562 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:05,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:05,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:05,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:05,562 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:05,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:05,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:05,562 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:05,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:05,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:05,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:05,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:05,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:05,563 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:05,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:05,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:05,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,565 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,565 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:05,565 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:05,565 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,565 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:05,565 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:05,565 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:05,566 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:05,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:05,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:05,566 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:05,566 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:05,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:05,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:05,566 DEBUG [Thread-875] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:05,566 DEBUG [Thread-875] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:05,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:05,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:05,566 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:57:05,666 DEBUG [Thread-875] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:05,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:05,668 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:05,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:05,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:05,670 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:05,670 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:05,670 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:05,670 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:05,670 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:05,670 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:05,671 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,671 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:05,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:05,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:05,672 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,672 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:05,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:05,673 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:05,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:05,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:05,673 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:05,685 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:05,686 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:05,686 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:05,686 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:05,686 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:05,686 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:05,686 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:05,686 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:57:05,686 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. started... 2023-03-20 10:57:05,686 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. as already flushing 2023-03-20 10:57:05,686 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:57:05,686 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:05,686 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:05,686 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,686 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:05,687 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:05,687 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,687 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:05,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:05,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:05,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:05,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:05,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:05,690 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:05,690 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:05,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:05,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:05,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18ed42d7[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:05,693 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:05,693 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:05,696 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,696 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:05,702 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:05,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:05,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:05,702 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:05,702 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:05,702 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:05,704 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,704 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:05,704 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:05,704 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:05,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:05,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:05,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:05,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:05,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:05,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:05,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:05,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:05,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:05,710 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:05,710 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:05,710 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:05,710 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:05,710 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:05,712 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:05,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:05,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:05,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:05,713 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:05,713 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:05,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:05,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:05,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:05,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:05,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:05,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:05,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:05,714 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:05,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:05,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,716 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:05,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:05,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,721 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:05,721 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:05,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:05,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:05,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:05,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:05,722 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:05,722 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,722 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:05,722 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:05,722 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:05,722 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:57:05,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:05,723 DEBUG [Thread-875] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:05,723 DEBUG [Thread-875] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:05,722 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:05,723 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:05,723 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:05,794 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=362 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/C/7cd4503322ff4fb1b9857f465c4ceb12 2023-03-20 10:57:05,806 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/A/48549e470b2c4c119765fca40fbff98c as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/48549e470b2c4c119765fca40fbff98c 2023-03-20 10:57:05,813 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/48549e470b2c4c119765fca40fbff98c, entries=200, sequenceid=362, filesize=32.3 K 2023-03-20 10:57:05,814 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/B/017afb26562c4e0894f44f1ab1bbc8fb as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/017afb26562c4e0894f44f1ab1bbc8fb 2023-03-20 10:57:05,821 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/017afb26562c4e0894f44f1ab1bbc8fb, entries=150, sequenceid=362, filesize=11.9 K 2023-03-20 10:57:05,823 DEBUG [Thread-875] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:05,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:05,828 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/C/7cd4503322ff4fb1b9857f465c4ceb12 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/7cd4503322ff4fb1b9857f465c4ceb12 2023-03-20 10:57:05,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:05,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:05,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:05,840 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:05,840 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:05,840 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:05,840 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:05,840 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:05,840 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:05,841 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:05,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:05,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:05,841 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,841 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,842 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:05,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:05,842 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:05,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:05,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:05,842 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:05,847 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:05,847 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:05,847 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:05,847 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:05,847 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:05,847 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:05,847 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:05,847 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:57:05,847 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. started... 2023-03-20 10:57:05,848 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. as already flushing 2023-03-20 10:57:05,848 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:57:05,848 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:05,848 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:05,848 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,848 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:05,848 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:05,849 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/7cd4503322ff4fb1b9857f465c4ceb12, entries=150, sequenceid=362, filesize=11.9 K 2023-03-20 10:57:05,850 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 9fc4d04a6b2183ce1217c4847d39aded in 971ms, sequenceid=362, compaction requested=true 2023-03-20 10:57:05,850 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9fc4d04a6b2183ce1217c4847d39aded: 2023-03-20 10:57:05,850 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:57:05,850 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 10:57:05,850 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 10:57:05,850 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:57:05,851 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:57:05,853 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:05,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:05,853 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:05,854 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:05,854 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:05,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:05,855 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49324 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 10:57:05,855 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): 9fc4d04a6b2183ce1217c4847d39aded/B is initiating minor compaction (all files) 2023-03-20 10:57:05,855 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9fc4d04a6b2183ce1217c4847d39aded/B in TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:57:05,855 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/e327e36297f34a908a080974c7c58f8b, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/f8e15d6f47c0408bac3066fa053a54fc, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/00fd09649c1344c09dff2e52e338d56a, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/017afb26562c4e0894f44f1ab1bbc8fb] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp, totalSize=48.2 K 2023-03-20 10:57:05,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:05,856 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 111804 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 10:57:05,856 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1912): 9fc4d04a6b2183ce1217c4847d39aded/A is initiating minor compaction (all files) 2023-03-20 10:57:05,856 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9fc4d04a6b2183ce1217c4847d39aded/A in TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:57:05,856 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/9391e6713cce47cd8ddff9b6c3f39905, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/c9d269d945524ba2a12f5fa5c4b79d8c, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/8eecf89b87674f768f83df4fa1a9fec5, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/48549e470b2c4c119765fca40fbff98c] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp, totalSize=109.2 K 2023-03-20 10:57:05,856 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting e327e36297f34a908a080974c7c58f8b, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=313, earliestPutTs=1679309823181 2023-03-20 10:57:05,857 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 9391e6713cce47cd8ddff9b6c3f39905, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=313, earliestPutTs=1679309823181 2023-03-20 10:57:05,857 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting f8e15d6f47c0408bac3066fa053a54fc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=327, earliestPutTs=1679309823204 2023-03-20 10:57:05,857 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting c9d269d945524ba2a12f5fa5c4b79d8c, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=327, earliestPutTs=1679309823204 2023-03-20 10:57:05,857 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 00fd09649c1344c09dff2e52e338d56a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=349, earliestPutTs=1679309823566 2023-03-20 10:57:05,858 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 8eecf89b87674f768f83df4fa1a9fec5, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=349, earliestPutTs=1679309823566 2023-03-20 10:57:05,858 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 48549e470b2c4c119765fca40fbff98c, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1679309824242 2023-03-20 10:57:05,858 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 017afb26562c4e0894f44f1ab1bbc8fb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1679309824242 2023-03-20 10:57:05,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:05,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:05,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:05,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@464bc7cd[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:05,869 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:05,869 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:05,871 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:05,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:05,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:05,871 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:05,871 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:05,871 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:05,873 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:05,873 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:05,873 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:05,873 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,873 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:05,873 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:05,873 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:05,873 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:05,873 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:05,873 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:05,873 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:05,874 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:05,874 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:05,875 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:05,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:05,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:05,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:05,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:05,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:05,875 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:05,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:05,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:05,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:05,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:05,876 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:05,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:05,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:05,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:05,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,880 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:05,880 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:05,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:05,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:05,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:05,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:05,881 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,881 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:05,881 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:05,881 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:05,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:05,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:05,881 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:05,881 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:05,881 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:05,881 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:05,882 DEBUG [Thread-875] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:05,882 DEBUG [Thread-875] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:05,915 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): 9fc4d04a6b2183ce1217c4847d39aded#B#compaction#175 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:57:05,948 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9fc4d04a6b2183ce1217c4847d39aded#A#compaction#174 average throughput is 0.44 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:57:05,982 DEBUG [Thread-875] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:05,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:05,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:06,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:06,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:06,003 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:06,003 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:06,003 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:06,003 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:06,003 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:06,003 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:06,004 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:06,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:06,004 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:06,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:06,004 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:06,005 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:06,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:06,005 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:06,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:06,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:06,005 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:06,010 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:06,010 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:06,010 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:06,010 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:06,010 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:06,010 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:06,010 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:06,011 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:57:06,011 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. started... 2023-03-20 10:57:06,012 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9fc4d04a6b2183ce1217c4847d39aded 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-20 10:57:06,012 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9fc4d04a6b2183ce1217c4847d39aded, store=A 2023-03-20 10:57:06,012 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:06,012 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9fc4d04a6b2183ce1217c4847d39aded, store=B 2023-03-20 10:57:06,012 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:06,012 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9fc4d04a6b2183ce1217c4847d39aded, store=C 2023-03-20 10:57:06,012 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:06,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 9fc4d04a6b2183ce1217c4847d39aded 2023-03-20 10:57:06,052 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. as already flushing 2023-03-20 10:57:06,112 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:06,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42958 deadline: 1679309886107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:06,115 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:06,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42936 deadline: 1679309886113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:06,217 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:06,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42958 deadline: 1679309886213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:06,219 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:06,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42936 deadline: 1679309886216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:06,271 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:06,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43002 deadline: 1679309886266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:06,273 DEBUG [Thread-871] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18253 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded., hostname=jenkins-hbase20.apache.org,36041,1679309777062, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-20 10:57:06,377 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/B/9191bf5214d249b1bf51fdb67b677b84 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/9191bf5214d249b1bf51fdb67b677b84 2023-03-20 10:57:06,383 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9fc4d04a6b2183ce1217c4847d39aded/B of 9fc4d04a6b2183ce1217c4847d39aded into 9191bf5214d249b1bf51fdb67b677b84(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:57:06,383 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9fc4d04a6b2183ce1217c4847d39aded: 2023-03-20 10:57:06,383 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded., storeName=9fc4d04a6b2183ce1217c4847d39aded/B, priority=12, startTime=1679309825850; duration=0sec 2023-03-20 10:57:06,383 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:57:06,383 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 10:57:06,386 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49324 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 10:57:06,386 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): 9fc4d04a6b2183ce1217c4847d39aded/C is initiating minor compaction (all files) 2023-03-20 10:57:06,386 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9fc4d04a6b2183ce1217c4847d39aded/C in TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:57:06,387 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/89f232e1b943419cbf08edf2229a6095, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/6fdd90dc469941cbb126702523ea3a18, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/e40adb40dbb04e68b5c84c789ec66876, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/7cd4503322ff4fb1b9857f465c4ceb12] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp, totalSize=48.2 K 2023-03-20 10:57:06,387 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 89f232e1b943419cbf08edf2229a6095, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=313, earliestPutTs=1679309823181 2023-03-20 10:57:06,388 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 6fdd90dc469941cbb126702523ea3a18, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=327, earliestPutTs=1679309823204 2023-03-20 10:57:06,388 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting e40adb40dbb04e68b5c84c789ec66876, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=349, earliestPutTs=1679309823566 2023-03-20 10:57:06,388 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 7cd4503322ff4fb1b9857f465c4ceb12, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1679309824242 2023-03-20 10:57:06,397 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): 9fc4d04a6b2183ce1217c4847d39aded#C#compaction#177 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:57:06,421 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:06,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42936 deadline: 1679309886420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:06,422 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:06,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42958 deadline: 1679309886418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:06,469 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/C/731a4ee53a294a539af9c7c93908d796 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/731a4ee53a294a539af9c7c93908d796 2023-03-20 10:57:06,476 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230320252afdc6d4644375ad3b00b9ed24306c to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320252afdc6d4644375ad3b00b9ed24306c 2023-03-20 10:57:06,479 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9fc4d04a6b2183ce1217c4847d39aded/C of 9fc4d04a6b2183ce1217c4847d39aded into 731a4ee53a294a539af9c7c93908d796(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:57:06,479 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9fc4d04a6b2183ce1217c4847d39aded: 2023-03-20 10:57:06,479 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded., storeName=9fc4d04a6b2183ce1217c4847d39aded/C, priority=12, startTime=1679309825851; duration=0sec 2023-03-20 10:57:06,479 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:57:06,724 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:06,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42936 deadline: 1679309886723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:06,724 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:06,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42958 deadline: 1679309886723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:06,873 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/A/4cc0c745ae3742ac8b68f2c64fd2036e as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/4cc0c745ae3742ac8b68f2c64fd2036e 2023-03-20 10:57:06,886 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=381, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/A/83c25e9d201647c2832d3fec03f16c93 2023-03-20 10:57:06,904 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9fc4d04a6b2183ce1217c4847d39aded/A of 9fc4d04a6b2183ce1217c4847d39aded into 4cc0c745ae3742ac8b68f2c64fd2036e(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-03-20 10:57:06,904 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9fc4d04a6b2183ce1217c4847d39aded: 2023-03-20 10:57:06,904 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded., storeName=9fc4d04a6b2183ce1217c4847d39aded/A, priority=12, startTime=1679309825850; duration=1sec 2023-03-20 10:57:06,904 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:57:06,927 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=381 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/B/f5219f6e34864531850d4f2dab475e01 2023-03-20 10:57:06,970 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=381 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/C/e7b19404c262479c8b6aa054dfd3b93d 2023-03-20 10:57:06,984 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/A/83c25e9d201647c2832d3fec03f16c93 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/83c25e9d201647c2832d3fec03f16c93 2023-03-20 10:57:06,989 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/83c25e9d201647c2832d3fec03f16c93, entries=150, sequenceid=381, filesize=25.4 K 2023-03-20 10:57:06,995 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/B/f5219f6e34864531850d4f2dab475e01 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/f5219f6e34864531850d4f2dab475e01 2023-03-20 10:57:07,010 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/f5219f6e34864531850d4f2dab475e01, entries=150, sequenceid=381, filesize=11.9 K 2023-03-20 10:57:07,012 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/C/e7b19404c262479c8b6aa054dfd3b93d as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/e7b19404c262479c8b6aa054dfd3b93d 2023-03-20 10:57:07,017 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/e7b19404c262479c8b6aa054dfd3b93d, entries=150, sequenceid=381, filesize=11.9 K 2023-03-20 10:57:07,019 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 9fc4d04a6b2183ce1217c4847d39aded in 1007ms, sequenceid=381, compaction requested=false 2023-03-20 10:57:07,020 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9fc4d04a6b2183ce1217c4847d39aded: 2023-03-20 10:57:07,020 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:57:07,020 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:07,020 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:07,020 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:07,022 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:07,022 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:07,025 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:07,025 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:07,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:07,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:07,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:07,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:07,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:07,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:07,026 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:07,026 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:07,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:07,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:07,028 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:07,028 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d5c6309[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:07,028 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:07,029 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:07,030 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:07,030 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:07,031 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:07,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:07,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:07,031 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:07,031 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:07,031 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:07,035 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:07,035 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:07,035 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:07,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:07,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:07,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:07,036 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:07,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:07,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:07,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:07,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:07,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:07,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:07,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:07,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:07,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:07,037 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:07,037 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:07,037 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:07,037 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:07,038 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:07,039 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:07,039 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:07,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:07,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:07,039 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:07,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:07,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:07,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:07,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:07,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:07,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:07,040 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:07,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:07,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:07,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:07,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:07,046 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:07,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:07,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:07,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:07,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:07,063 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:07,063 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:07,063 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:07,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:07,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:07,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:07,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:07,063 DEBUG [Thread-875] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:07,064 DEBUG [Thread-875] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:07,063 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:07,064 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:07,064 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:07,064 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:07,064 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:07,064 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:07,064 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:07,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:07,065 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:57:07,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:07,164 DEBUG [Thread-875] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:07,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:07,172 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:07,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:07,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:07,173 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:07,174 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:07,174 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:07,174 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:07,174 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:07,174 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:07,175 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:07,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:07,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:07,175 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:07,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:07,175 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:07,175 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:07,176 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:07,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:07,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:07,176 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:07,177 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:07,178 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:07,178 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:07,178 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:07,178 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:07,182 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:07,182 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:07,182 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:57:07,182 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. started... 2023-03-20 10:57:07,188 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9fc4d04a6b2183ce1217c4847d39aded 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-20 10:57:07,188 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9fc4d04a6b2183ce1217c4847d39aded, store=A 2023-03-20 10:57:07,188 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:07,188 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9fc4d04a6b2183ce1217c4847d39aded, store=B 2023-03-20 10:57:07,188 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:07,188 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9fc4d04a6b2183ce1217c4847d39aded, store=C 2023-03-20 10:57:07,188 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:07,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 9fc4d04a6b2183ce1217c4847d39aded 2023-03-20 10:57:07,233 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. as already flushing 2023-03-20 10:57:07,238 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230320a04173d0920c43ad97938e352fc9cf4c to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320a04173d0920c43ad97938e352fc9cf4c 2023-03-20 10:57:07,264 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=397, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/A/7e47de709e5242db8a31cb690678fb26 2023-03-20 10:57:07,281 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:07,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42936 deadline: 1679309887276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:07,281 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:07,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42958 deadline: 1679309887276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:07,387 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:07,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42936 deadline: 1679309887384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:07,388 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:07,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42958 deadline: 1679309887385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:07,590 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:07,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42958 deadline: 1679309887590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:07,590 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:07,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:42936 deadline: 1679309887590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:07,733 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=397 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/B/584c993ee7ca432a808d4fade8abe489 2023-03-20 10:57:07,788 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=397 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/C/de4a2f70f29045dd90984e48b1938c47 2023-03-20 10:57:07,800 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/A/7e47de709e5242db8a31cb690678fb26 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/7e47de709e5242db8a31cb690678fb26 2023-03-20 10:57:07,809 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/7e47de709e5242db8a31cb690678fb26, entries=150, sequenceid=397, filesize=25.4 K 2023-03-20 10:57:07,810 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/B/584c993ee7ca432a808d4fade8abe489 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/584c993ee7ca432a808d4fade8abe489 2023-03-20 10:57:07,816 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/584c993ee7ca432a808d4fade8abe489, entries=150, sequenceid=397, filesize=11.9 K 2023-03-20 10:57:07,817 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/C/de4a2f70f29045dd90984e48b1938c47 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/de4a2f70f29045dd90984e48b1938c47 2023-03-20 10:57:07,823 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/de4a2f70f29045dd90984e48b1938c47, entries=150, sequenceid=397, filesize=11.9 K 2023-03-20 10:57:07,824 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 9fc4d04a6b2183ce1217c4847d39aded in 636ms, sequenceid=397, compaction requested=true 2023-03-20 10:57:07,824 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9fc4d04a6b2183ce1217c4847d39aded: 2023-03-20 10:57:07,824 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:57:07,824 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:07,824 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:07,824 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:07,824 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:07,824 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:07,826 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:07,826 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:07,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:07,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:07,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:07,826 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:07,826 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:07,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:07,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:07,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:07,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:07,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:07,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:07,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56fe37f6[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:07,828 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:07,829 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:07,829 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:07,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:07,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:07,829 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:07,829 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:07,829 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:07,829 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:07,829 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:07,830 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:07,830 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:07,830 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:07,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:07,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:07,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:07,830 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:07,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:07,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:07,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:07,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:07,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:07,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:07,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:07,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:07,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:07,833 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:07,833 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:07,833 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:07,833 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:07,833 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:07,834 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:07,835 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:07,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:07,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:07,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:07,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:07,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:07,836 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:07,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:07,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:07,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:07,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:07,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:07,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:07,840 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:07,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:07,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:07,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:07,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:07,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:07,843 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:07,846 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:07,846 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:07,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:07,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:07,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:07,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:07,847 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:07,847 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:07,847 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:07,847 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:07,847 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:07,847 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:07,847 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:07,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:07,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:07,848 DEBUG [Thread-875] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:07,848 DEBUG [Thread-875] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:07,847 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:07,899 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9fc4d04a6b2183ce1217c4847d39aded 3/3 column families, dataSize=114.05 KB heapSize=339.66 KB 2023-03-20 10:57:07,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 9fc4d04a6b2183ce1217c4847d39aded 2023-03-20 10:57:07,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679309827245, max timestamp=1679309827897Num uniques 859; 2023-03-20 10:57:07,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679309827245, max timestamp=1679309827897Num uniques 859; 2023-03-20 10:57:07,901 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-20 10:57:07,900 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679309827245, max timestamp=1679309827897Num uniques 859; 2023-03-20 10:57:07,899 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9fc4d04a6b2183ce1217c4847d39aded, store=A 2023-03-20 10:57:07,901 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-20 10:57:07,901 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-20 10:57:07,901 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:57:07,901 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9fc4d04a6b2183ce1217c4847d39aded, store=B 2023-03-20 10:57:07,901 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:57:07,901 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9fc4d04a6b2183ce1217c4847d39aded, store=C 2023-03-20 10:57:07,901 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:57:07,904 DEBUG [Thread-876] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2c411e08 to 127.0.0.1:60226 2023-03-20 10:57:07,904 DEBUG [Thread-876] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 10:57:07,906 DEBUG [Thread-878] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7605c204 to 127.0.0.1:60226 2023-03-20 10:57:07,906 DEBUG [Thread-878] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 10:57:07,909 DEBUG [Thread-869] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x633c81ba to 127.0.0.1:60226 2023-03-20 10:57:07,909 DEBUG [Thread-869] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 10:57:07,909 DEBUG [Thread-873] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x43a14bcf to 127.0.0.1:60226 2023-03-20 10:57:07,909 DEBUG [Thread-873] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 10:57:07,910 DEBUG [Thread-880] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x16505096 to 127.0.0.1:60226 2023-03-20 10:57:07,910 DEBUG [Thread-880] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 10:57:07,911 DEBUG [Thread-882] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x03e1990e to 127.0.0.1:60226 2023-03-20 10:57:07,911 DEBUG [Thread-882] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 10:57:07,948 DEBUG [Thread-875] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:07,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:07,987 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303205136c2306a9048e08eb592be96eff8c4 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303205136c2306a9048e08eb592be96eff8c4 2023-03-20 10:57:08,026 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=418, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/A/210e59a3f7124fc58fe8d6ccd35308a7 2023-03-20 10:57:08,453 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=418 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/B/217572ca530b4b4b8e75a644fd7fcacf 2023-03-20 10:57:08,896 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=418 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/C/220d76e5a306467ca44e464c337a8655 2023-03-20 10:57:08,900 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/A/210e59a3f7124fc58fe8d6ccd35308a7 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/210e59a3f7124fc58fe8d6ccd35308a7 2023-03-20 10:57:08,904 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/210e59a3f7124fc58fe8d6ccd35308a7, entries=250, sequenceid=418, filesize=39.2 K 2023-03-20 10:57:08,905 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/B/217572ca530b4b4b8e75a644fd7fcacf as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/217572ca530b4b4b8e75a644fd7fcacf 2023-03-20 10:57:08,911 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/217572ca530b4b4b8e75a644fd7fcacf, entries=150, sequenceid=418, filesize=11.9 K 2023-03-20 10:57:08,912 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/C/220d76e5a306467ca44e464c337a8655 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/220d76e5a306467ca44e464c337a8655 2023-03-20 10:57:08,918 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/220d76e5a306467ca44e464c337a8655, entries=150, sequenceid=418, filesize=11.9 K 2023-03-20 10:57:08,919 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~81.42 KB/83376, currentSize=13.42 KB/13740 for 9fc4d04a6b2183ce1217c4847d39aded in 1020ms, sequenceid=418, compaction requested=true 2023-03-20 10:57:08,919 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9fc4d04a6b2183ce1217c4847d39aded: 2023-03-20 10:57:08,919 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:57:08,919 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 10:57:08,919 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 10:57:08,919 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:57:08,919 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:57:08,920 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 118984 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 10:57:08,920 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49460 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 10:57:08,921 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1912): 9fc4d04a6b2183ce1217c4847d39aded/A is initiating minor compaction (all files) 2023-03-20 10:57:08,921 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): 9fc4d04a6b2183ce1217c4847d39aded/B is initiating minor compaction (all files) 2023-03-20 10:57:08,921 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9fc4d04a6b2183ce1217c4847d39aded/A in TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:57:08,921 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9fc4d04a6b2183ce1217c4847d39aded/B in TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:57:08,921 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/4cc0c745ae3742ac8b68f2c64fd2036e, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/83c25e9d201647c2832d3fec03f16c93, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/7e47de709e5242db8a31cb690678fb26, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/210e59a3f7124fc58fe8d6ccd35308a7] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp, totalSize=116.2 K 2023-03-20 10:57:08,921 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/9191bf5214d249b1bf51fdb67b677b84, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/f5219f6e34864531850d4f2dab475e01, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/584c993ee7ca432a808d4fade8abe489, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/217572ca530b4b4b8e75a644fd7fcacf] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp, totalSize=48.3 K 2023-03-20 10:57:08,921 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 4cc0c745ae3742ac8b68f2c64fd2036e, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1679309824242 2023-03-20 10:57:08,921 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 9191bf5214d249b1bf51fdb67b677b84, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1679309824242 2023-03-20 10:57:08,921 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 83c25e9d201647c2832d3fec03f16c93, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=381, earliestPutTs=1679309824910 2023-03-20 10:57:08,921 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting f5219f6e34864531850d4f2dab475e01, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=381, earliestPutTs=1679309824910 2023-03-20 10:57:08,922 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 7e47de709e5242db8a31cb690678fb26, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=397, earliestPutTs=1679309826058 2023-03-20 10:57:08,922 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 584c993ee7ca432a808d4fade8abe489, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=397, earliestPutTs=1679309826058 2023-03-20 10:57:08,922 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 210e59a3f7124fc58fe8d6ccd35308a7, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=418, earliestPutTs=1679309827245 2023-03-20 10:57:08,922 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 217572ca530b4b4b8e75a644fd7fcacf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=418, earliestPutTs=1679309827245 2023-03-20 10:57:08,935 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): 9fc4d04a6b2183ce1217c4847d39aded#B#compaction#186 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:57:08,951 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9fc4d04a6b2183ce1217c4847d39aded#A#compaction#187 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:57:08,955 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/B/6487b92b8c924dd48e0c3eb922919434 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/6487b92b8c924dd48e0c3eb922919434 2023-03-20 10:57:08,960 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9fc4d04a6b2183ce1217c4847d39aded/B of 9fc4d04a6b2183ce1217c4847d39aded into 6487b92b8c924dd48e0c3eb922919434(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:57:08,960 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9fc4d04a6b2183ce1217c4847d39aded: 2023-03-20 10:57:08,960 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded., storeName=9fc4d04a6b2183ce1217c4847d39aded/B, priority=12, startTime=1679309828919; duration=0sec 2023-03-20 10:57:08,960 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:57:08,961 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 10:57:08,962 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49460 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 10:57:08,962 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): 9fc4d04a6b2183ce1217c4847d39aded/C is initiating minor compaction (all files) 2023-03-20 10:57:08,962 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9fc4d04a6b2183ce1217c4847d39aded/C in TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:57:08,962 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/731a4ee53a294a539af9c7c93908d796, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/e7b19404c262479c8b6aa054dfd3b93d, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/de4a2f70f29045dd90984e48b1938c47, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/220d76e5a306467ca44e464c337a8655] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp, totalSize=48.3 K 2023-03-20 10:57:08,962 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 731a4ee53a294a539af9c7c93908d796, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1679309824242 2023-03-20 10:57:08,963 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting e7b19404c262479c8b6aa054dfd3b93d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=381, earliestPutTs=1679309824910 2023-03-20 10:57:08,963 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting de4a2f70f29045dd90984e48b1938c47, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=397, earliestPutTs=1679309826058 2023-03-20 10:57:08,963 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 220d76e5a306467ca44e464c337a8655, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=418, earliestPutTs=1679309827245 2023-03-20 10:57:08,979 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): 9fc4d04a6b2183ce1217c4847d39aded#C#compaction#188 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:57:09,001 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/C/fe57225e4fb5439f8917f5ad61a0bdfb as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/fe57225e4fb5439f8917f5ad61a0bdfb 2023-03-20 10:57:09,011 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9fc4d04a6b2183ce1217c4847d39aded/C of 9fc4d04a6b2183ce1217c4847d39aded into fe57225e4fb5439f8917f5ad61a0bdfb(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:57:09,011 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9fc4d04a6b2183ce1217c4847d39aded: 2023-03-20 10:57:09,011 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded., storeName=9fc4d04a6b2183ce1217c4847d39aded/C, priority=12, startTime=1679309828919; duration=0sec 2023-03-20 10:57:09,012 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:57:09,013 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/A/bee2e18c02964f8394b8ac02c527f2be as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/bee2e18c02964f8394b8ac02c527f2be 2023-03-20 10:57:09,020 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9fc4d04a6b2183ce1217c4847d39aded/A of 9fc4d04a6b2183ce1217c4847d39aded into bee2e18c02964f8394b8ac02c527f2be(size=26.3 K), total size for store is 26.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:57:09,020 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9fc4d04a6b2183ce1217c4847d39aded: 2023-03-20 10:57:09,020 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded., storeName=9fc4d04a6b2183ce1217c4847d39aded/A, priority=12, startTime=1679309828919; duration=0sec 2023-03-20 10:57:09,020 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:57:11,241 DEBUG [Thread-865] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3b75e45b to 127.0.0.1:60226 2023-03-20 10:57:11,241 DEBUG [Thread-865] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 10:57:13,316 DEBUG [Thread-867] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6017755a to 127.0.0.1:60226 2023-03-20 10:57:13,316 DEBUG [Thread-867] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 10:57:16,371 DEBUG [Thread-871] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x661b21c2 to 127.0.0.1:60226 2023-03-20 10:57:16,372 DEBUG [Thread-871] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 10:57:16,372 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-20 10:57:16,372 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(394): wrote 26 2023-03-20 10:57:16,372 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(394): wrote 22 2023-03-20 10:57:16,372 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(394): wrote 146 2023-03-20 10:57:16,372 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(394): wrote 8 2023-03-20 10:57:16,372 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(394): wrote 142 2023-03-20 10:57:16,372 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-20 10:57:16,372 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(398): read 3396 2023-03-20 10:57:16,372 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(398): read 3387 2023-03-20 10:57:16,372 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-20 10:57:16,372 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(402): scanned 1465 2023-03-20 10:57:16,372 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(403): verified 4394 rows 2023-03-20 10:57:16,372 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(402): scanned 1457 2023-03-20 10:57:16,372 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(403): verified 4371 rows 2023-03-20 10:57:16,372 INFO [Listener at localhost.localdomain/43509] client.ConnectionImplementation(1959): Closing master protocol: MasterService 2023-03-20 10:57:16,372 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x65f279d2 to 127.0.0.1:60226 2023-03-20 10:57:16,372 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 10:57:16,374 INFO [Listener at localhost.localdomain/43509] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-20 10:57:16,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-03-20 10:57:16,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-20 10:57:16,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-20 10:57:16,381 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679309836380"}]},"ts":"1679309836380"} 2023-03-20 10:57:16,382 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-20 10:57:16,387 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-20 10:57:16,388 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=9fc4d04a6b2183ce1217c4847d39aded, UNASSIGN}] 2023-03-20 10:57:16,389 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=9fc4d04a6b2183ce1217c4847d39aded, UNASSIGN 2023-03-20 10:57:16,390 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=9fc4d04a6b2183ce1217c4847d39aded, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:16,390 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679309836390"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679309836390"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679309836390"}]},"ts":"1679309836390"} 2023-03-20 10:57:16,391 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; CloseRegionProcedure 9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062}] 2023-03-20 10:57:16,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-20 10:57:16,543 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 9fc4d04a6b2183ce1217c4847d39aded 2023-03-20 10:57:16,543 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 9fc4d04a6b2183ce1217c4847d39aded, disabling compactions & flushes 2023-03-20 10:57:16,543 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:57:16,543 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:57:16,543 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. after waiting 0 ms 2023-03-20 10:57:16,543 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:57:16,543 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 9fc4d04a6b2183ce1217c4847d39aded 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-03-20 10:57:16,544 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9fc4d04a6b2183ce1217c4847d39aded, store=A 2023-03-20 10:57:16,544 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:16,544 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9fc4d04a6b2183ce1217c4847d39aded, store=B 2023-03-20 10:57:16,544 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:16,544 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9fc4d04a6b2183ce1217c4847d39aded, store=C 2023-03-20 10:57:16,544 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:16,560 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230320427640bc4d4643888e0bd1f2ae9e5f49 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320427640bc4d4643888e0bd1f2ae9e5f49 2023-03-20 10:57:16,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-20 10:57:16,971 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=429, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/A/67a2c2119ed44dbeb5005cf4283b052c 2023-03-20 10:57:16,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-20 10:57:17,383 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=429 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/B/cf399f919455414db29d8bc1139304d0 2023-03-20 10:57:17,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-20 10:57:17,797 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=429 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/C/53ed99f4dfdf4ce299806d594499a853 2023-03-20 10:57:17,801 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/A/67a2c2119ed44dbeb5005cf4283b052c as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/67a2c2119ed44dbeb5005cf4283b052c 2023-03-20 10:57:17,806 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/67a2c2119ed44dbeb5005cf4283b052c, entries=150, sequenceid=429, filesize=25.4 K 2023-03-20 10:57:17,806 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/B/cf399f919455414db29d8bc1139304d0 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/cf399f919455414db29d8bc1139304d0 2023-03-20 10:57:17,810 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/cf399f919455414db29d8bc1139304d0, entries=150, sequenceid=429, filesize=11.9 K 2023-03-20 10:57:17,811 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/.tmp/C/53ed99f4dfdf4ce299806d594499a853 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/53ed99f4dfdf4ce299806d594499a853 2023-03-20 10:57:17,815 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/53ed99f4dfdf4ce299806d594499a853, entries=150, sequenceid=429, filesize=11.9 K 2023-03-20 10:57:17,816 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for 9fc4d04a6b2183ce1217c4847d39aded in 1273ms, sequenceid=429, compaction requested=false 2023-03-20 10:57:17,833 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/72f846aad3344212adb40a49efd0b339, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/803bce3658e94a5bbf95a9784858871e, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/8be7d7523828421c88108f18d94b744b, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/9c31b8bb7b964321af7a3300a48b0852, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/df09ecfeda5f41758eca14656efe0490, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/68564e81a07c46789d63bd6b483206a0, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/dc6177d1b44449119f504531a2618b47, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/f62fd68cef9c44e4a448d7317e332f61, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/89fe1a698b064675a497fefe7775f72e, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/9c4d50edf211489fb2e8c8d9ca86e846, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/6e630386549944c0bd784fcc3a6a120a, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/c2dfa925447f4b4286a190f423e5ccf5, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/cb94e2f9fbb94325a40f6784aeb23e94, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/bf240f17c7314b16bf37781a36bbcc5a, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/d0ce449621844463a5b8566e859a72da, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/70e74d1d86e4431fa0adc701a4a9a589, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/b678fc7564704a5a8771a284bcfea8eb, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/9391e6713cce47cd8ddff9b6c3f39905, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/c9d269d945524ba2a12f5fa5c4b79d8c, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/8eecf89b87674f768f83df4fa1a9fec5, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/48549e470b2c4c119765fca40fbff98c, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/4cc0c745ae3742ac8b68f2c64fd2036e, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/83c25e9d201647c2832d3fec03f16c93, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/7e47de709e5242db8a31cb690678fb26, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/210e59a3f7124fc58fe8d6ccd35308a7] to archive 2023-03-20 10:57:17,834 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-20 10:57:17,835 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/72f846aad3344212adb40a49efd0b339 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/72f846aad3344212adb40a49efd0b339 2023-03-20 10:57:17,837 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/803bce3658e94a5bbf95a9784858871e to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/803bce3658e94a5bbf95a9784858871e 2023-03-20 10:57:17,838 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/8be7d7523828421c88108f18d94b744b to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/8be7d7523828421c88108f18d94b744b 2023-03-20 10:57:17,839 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/9c31b8bb7b964321af7a3300a48b0852 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/9c31b8bb7b964321af7a3300a48b0852 2023-03-20 10:57:17,840 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/df09ecfeda5f41758eca14656efe0490 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/df09ecfeda5f41758eca14656efe0490 2023-03-20 10:57:17,841 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/68564e81a07c46789d63bd6b483206a0 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/68564e81a07c46789d63bd6b483206a0 2023-03-20 10:57:17,843 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/dc6177d1b44449119f504531a2618b47 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/dc6177d1b44449119f504531a2618b47 2023-03-20 10:57:17,844 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/f62fd68cef9c44e4a448d7317e332f61 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/f62fd68cef9c44e4a448d7317e332f61 2023-03-20 10:57:17,845 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/89fe1a698b064675a497fefe7775f72e to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/89fe1a698b064675a497fefe7775f72e 2023-03-20 10:57:17,846 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/9c4d50edf211489fb2e8c8d9ca86e846 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/9c4d50edf211489fb2e8c8d9ca86e846 2023-03-20 10:57:17,847 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/6e630386549944c0bd784fcc3a6a120a to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/6e630386549944c0bd784fcc3a6a120a 2023-03-20 10:57:17,848 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/c2dfa925447f4b4286a190f423e5ccf5 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/c2dfa925447f4b4286a190f423e5ccf5 2023-03-20 10:57:17,849 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/cb94e2f9fbb94325a40f6784aeb23e94 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/cb94e2f9fbb94325a40f6784aeb23e94 2023-03-20 10:57:17,850 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/bf240f17c7314b16bf37781a36bbcc5a to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/bf240f17c7314b16bf37781a36bbcc5a 2023-03-20 10:57:17,851 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/d0ce449621844463a5b8566e859a72da to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/d0ce449621844463a5b8566e859a72da 2023-03-20 10:57:17,852 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/70e74d1d86e4431fa0adc701a4a9a589 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/70e74d1d86e4431fa0adc701a4a9a589 2023-03-20 10:57:17,853 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/b678fc7564704a5a8771a284bcfea8eb to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/b678fc7564704a5a8771a284bcfea8eb 2023-03-20 10:57:17,854 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/9391e6713cce47cd8ddff9b6c3f39905 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/9391e6713cce47cd8ddff9b6c3f39905 2023-03-20 10:57:17,855 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/c9d269d945524ba2a12f5fa5c4b79d8c to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/c9d269d945524ba2a12f5fa5c4b79d8c 2023-03-20 10:57:17,855 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/8eecf89b87674f768f83df4fa1a9fec5 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/8eecf89b87674f768f83df4fa1a9fec5 2023-03-20 10:57:17,856 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/48549e470b2c4c119765fca40fbff98c to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/48549e470b2c4c119765fca40fbff98c 2023-03-20 10:57:17,857 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/4cc0c745ae3742ac8b68f2c64fd2036e to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/4cc0c745ae3742ac8b68f2c64fd2036e 2023-03-20 10:57:17,858 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/83c25e9d201647c2832d3fec03f16c93 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/83c25e9d201647c2832d3fec03f16c93 2023-03-20 10:57:17,859 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/7e47de709e5242db8a31cb690678fb26 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/7e47de709e5242db8a31cb690678fb26 2023-03-20 10:57:17,860 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/210e59a3f7124fc58fe8d6ccd35308a7 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/210e59a3f7124fc58fe8d6ccd35308a7 2023-03-20 10:57:17,863 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/4583dacb48e043e79cce494d8b32e043, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/59de0e0eda424b9ca26438de8fca4f2f, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/6a68e3e657294e52a8fcc770a5f7df9b, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/8bfc066fd2c24d14a73c8c243b36c895, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/be09314619df44d3832029fe8762e5c6, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/4be62bbe615a418296b8f54c7e4ddc16, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/ae4c387285774dccaacee853fe0e2362, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/b6f5a113e7714d24988a7c0e862019c0, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/f17638332af0452aa0ff703c7c1828a9, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/e5e7adf94554422f908c7f6fb30dff3a, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/4ea0dfd6f05b42319bf06813fa611e2d, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/4d80978249f54a3696f1e8d053a7603f, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/2ff53b2e1b88479f93749e975bb249a8, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/0d8284b7d09b4719a3fa4f29bf1d4f7c, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/8ea7393145e94257934f33029fef2727, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/7f987e294b904f2196e5e9b95a76c5e8, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/e327e36297f34a908a080974c7c58f8b, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/c8b3401dc8db479086174beec11ed34b, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/f8e15d6f47c0408bac3066fa053a54fc, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/00fd09649c1344c09dff2e52e338d56a, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/9191bf5214d249b1bf51fdb67b677b84, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/017afb26562c4e0894f44f1ab1bbc8fb, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/f5219f6e34864531850d4f2dab475e01, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/584c993ee7ca432a808d4fade8abe489, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/217572ca530b4b4b8e75a644fd7fcacf] to archive 2023-03-20 10:57:17,864 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-20 10:57:17,866 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/4583dacb48e043e79cce494d8b32e043 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/4583dacb48e043e79cce494d8b32e043 2023-03-20 10:57:17,867 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/59de0e0eda424b9ca26438de8fca4f2f to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/59de0e0eda424b9ca26438de8fca4f2f 2023-03-20 10:57:17,868 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/6a68e3e657294e52a8fcc770a5f7df9b to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/6a68e3e657294e52a8fcc770a5f7df9b 2023-03-20 10:57:17,869 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/8bfc066fd2c24d14a73c8c243b36c895 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/8bfc066fd2c24d14a73c8c243b36c895 2023-03-20 10:57:17,870 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/be09314619df44d3832029fe8762e5c6 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/be09314619df44d3832029fe8762e5c6 2023-03-20 10:57:17,871 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/4be62bbe615a418296b8f54c7e4ddc16 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/4be62bbe615a418296b8f54c7e4ddc16 2023-03-20 10:57:17,872 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/ae4c387285774dccaacee853fe0e2362 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/ae4c387285774dccaacee853fe0e2362 2023-03-20 10:57:17,873 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/b6f5a113e7714d24988a7c0e862019c0 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/b6f5a113e7714d24988a7c0e862019c0 2023-03-20 10:57:17,874 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/f17638332af0452aa0ff703c7c1828a9 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/f17638332af0452aa0ff703c7c1828a9 2023-03-20 10:57:17,875 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/e5e7adf94554422f908c7f6fb30dff3a to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/e5e7adf94554422f908c7f6fb30dff3a 2023-03-20 10:57:17,876 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/4ea0dfd6f05b42319bf06813fa611e2d to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/4ea0dfd6f05b42319bf06813fa611e2d 2023-03-20 10:57:17,877 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/4d80978249f54a3696f1e8d053a7603f to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/4d80978249f54a3696f1e8d053a7603f 2023-03-20 10:57:17,878 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/2ff53b2e1b88479f93749e975bb249a8 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/2ff53b2e1b88479f93749e975bb249a8 2023-03-20 10:57:17,879 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/0d8284b7d09b4719a3fa4f29bf1d4f7c to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/0d8284b7d09b4719a3fa4f29bf1d4f7c 2023-03-20 10:57:17,880 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/8ea7393145e94257934f33029fef2727 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/8ea7393145e94257934f33029fef2727 2023-03-20 10:57:17,881 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/7f987e294b904f2196e5e9b95a76c5e8 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/7f987e294b904f2196e5e9b95a76c5e8 2023-03-20 10:57:17,881 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/e327e36297f34a908a080974c7c58f8b to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/e327e36297f34a908a080974c7c58f8b 2023-03-20 10:57:17,882 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/c8b3401dc8db479086174beec11ed34b to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/c8b3401dc8db479086174beec11ed34b 2023-03-20 10:57:17,883 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/f8e15d6f47c0408bac3066fa053a54fc to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/f8e15d6f47c0408bac3066fa053a54fc 2023-03-20 10:57:17,884 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/00fd09649c1344c09dff2e52e338d56a to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/00fd09649c1344c09dff2e52e338d56a 2023-03-20 10:57:17,885 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/9191bf5214d249b1bf51fdb67b677b84 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/9191bf5214d249b1bf51fdb67b677b84 2023-03-20 10:57:17,886 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/017afb26562c4e0894f44f1ab1bbc8fb to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/017afb26562c4e0894f44f1ab1bbc8fb 2023-03-20 10:57:17,887 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/f5219f6e34864531850d4f2dab475e01 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/f5219f6e34864531850d4f2dab475e01 2023-03-20 10:57:17,888 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/584c993ee7ca432a808d4fade8abe489 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/584c993ee7ca432a808d4fade8abe489 2023-03-20 10:57:17,889 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/217572ca530b4b4b8e75a644fd7fcacf to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/217572ca530b4b4b8e75a644fd7fcacf 2023-03-20 10:57:17,891 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/f8aa35048315471cb9227656cbc0b79b, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/368d499ef675406bbc45823c3716be61, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/c719bc090a4a49ed9f7725a2c3f864ab, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/700517f9411f4e8897bceb64a490f1f6, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/5a3487b49be6401a8ae673f5b5660673, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/d6be2ed31c414b00a6f5ba349909f7f2, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/7725125e02a840699e78029a30adff1d, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/ee90dee8ab6e4b9b9d080d19cffcec42, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/329cfc79a9a84b808f795f9f880cdded, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/9bdb3eb481744df19b69cf78e641b60e, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/b501de08c8b141b588706b7dfa423be2, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/99751ba587ef43dc8e1d7e873a1b721a, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/dd58f713e4954992a5600d4b2615b513, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/da5f6e9d88c54dd786d187d77b8c6b50, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/2c79da282a3245bfbc43a19cbeaba00e, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/597339b3b1d842eaa8d467c6d47ece72, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/89f232e1b943419cbf08edf2229a6095, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/7a39d3db07104287909c7e6aac06acf9, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/6fdd90dc469941cbb126702523ea3a18, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/e40adb40dbb04e68b5c84c789ec66876, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/731a4ee53a294a539af9c7c93908d796, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/7cd4503322ff4fb1b9857f465c4ceb12, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/e7b19404c262479c8b6aa054dfd3b93d, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/de4a2f70f29045dd90984e48b1938c47, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/220d76e5a306467ca44e464c337a8655] to archive 2023-03-20 10:57:17,892 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-20 10:57:17,893 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/f8aa35048315471cb9227656cbc0b79b to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/f8aa35048315471cb9227656cbc0b79b 2023-03-20 10:57:17,894 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/368d499ef675406bbc45823c3716be61 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/368d499ef675406bbc45823c3716be61 2023-03-20 10:57:17,895 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/c719bc090a4a49ed9f7725a2c3f864ab to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/c719bc090a4a49ed9f7725a2c3f864ab 2023-03-20 10:57:17,896 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/700517f9411f4e8897bceb64a490f1f6 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/700517f9411f4e8897bceb64a490f1f6 2023-03-20 10:57:17,897 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/5a3487b49be6401a8ae673f5b5660673 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/5a3487b49be6401a8ae673f5b5660673 2023-03-20 10:57:17,898 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/d6be2ed31c414b00a6f5ba349909f7f2 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/d6be2ed31c414b00a6f5ba349909f7f2 2023-03-20 10:57:17,899 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/7725125e02a840699e78029a30adff1d to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/7725125e02a840699e78029a30adff1d 2023-03-20 10:57:17,900 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/ee90dee8ab6e4b9b9d080d19cffcec42 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/ee90dee8ab6e4b9b9d080d19cffcec42 2023-03-20 10:57:17,901 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/329cfc79a9a84b808f795f9f880cdded to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/329cfc79a9a84b808f795f9f880cdded 2023-03-20 10:57:17,901 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/9bdb3eb481744df19b69cf78e641b60e to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/9bdb3eb481744df19b69cf78e641b60e 2023-03-20 10:57:17,902 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/b501de08c8b141b588706b7dfa423be2 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/b501de08c8b141b588706b7dfa423be2 2023-03-20 10:57:17,903 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/99751ba587ef43dc8e1d7e873a1b721a to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/99751ba587ef43dc8e1d7e873a1b721a 2023-03-20 10:57:17,904 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/dd58f713e4954992a5600d4b2615b513 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/dd58f713e4954992a5600d4b2615b513 2023-03-20 10:57:17,905 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/da5f6e9d88c54dd786d187d77b8c6b50 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/da5f6e9d88c54dd786d187d77b8c6b50 2023-03-20 10:57:17,906 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/2c79da282a3245bfbc43a19cbeaba00e to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/2c79da282a3245bfbc43a19cbeaba00e 2023-03-20 10:57:17,906 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/597339b3b1d842eaa8d467c6d47ece72 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/597339b3b1d842eaa8d467c6d47ece72 2023-03-20 10:57:17,907 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/89f232e1b943419cbf08edf2229a6095 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/89f232e1b943419cbf08edf2229a6095 2023-03-20 10:57:17,908 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/7a39d3db07104287909c7e6aac06acf9 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/7a39d3db07104287909c7e6aac06acf9 2023-03-20 10:57:17,909 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/6fdd90dc469941cbb126702523ea3a18 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/6fdd90dc469941cbb126702523ea3a18 2023-03-20 10:57:17,910 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/e40adb40dbb04e68b5c84c789ec66876 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/e40adb40dbb04e68b5c84c789ec66876 2023-03-20 10:57:17,912 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/731a4ee53a294a539af9c7c93908d796 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/731a4ee53a294a539af9c7c93908d796 2023-03-20 10:57:17,912 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/7cd4503322ff4fb1b9857f465c4ceb12 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/7cd4503322ff4fb1b9857f465c4ceb12 2023-03-20 10:57:17,913 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/e7b19404c262479c8b6aa054dfd3b93d to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/e7b19404c262479c8b6aa054dfd3b93d 2023-03-20 10:57:17,914 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/de4a2f70f29045dd90984e48b1938c47 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/de4a2f70f29045dd90984e48b1938c47 2023-03-20 10:57:17,915 DEBUG [StoreCloser-TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/220d76e5a306467ca44e464c337a8655 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/220d76e5a306467ca44e464c337a8655 2023-03-20 10:57:17,918 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/recovered.edits/432.seqid, newMaxSeqId=432, maxSeqId=4 2023-03-20 10:57:17,919 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded. 2023-03-20 10:57:17,920 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 9fc4d04a6b2183ce1217c4847d39aded: 2023-03-20 10:57:17,921 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 9fc4d04a6b2183ce1217c4847d39aded 2023-03-20 10:57:17,921 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=9fc4d04a6b2183ce1217c4847d39aded, regionState=CLOSED 2023-03-20 10:57:17,921 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679309837921"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679309837921"}]},"ts":"1679309837921"} 2023-03-20 10:57:17,923 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-03-20 10:57:17,923 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; CloseRegionProcedure 9fc4d04a6b2183ce1217c4847d39aded, server=jenkins-hbase20.apache.org,36041,1679309777062 in 1.5310 sec 2023-03-20 10:57:17,925 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-03-20 10:57:17,925 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=9fc4d04a6b2183ce1217c4847d39aded, UNASSIGN in 1.5350 sec 2023-03-20 10:57:17,925 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679309837925"}]},"ts":"1679309837925"} 2023-03-20 10:57:17,926 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-20 10:57:17,928 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-20 10:57:17,930 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5550 sec 2023-03-20 10:57:18,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-20 10:57:18,486 INFO [Listener at localhost.localdomain/43509] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 24 completed 2023-03-20 10:57:18,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-03-20 10:57:18,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-20 10:57:18,490 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-20 10:57:18,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-03-20 10:57:18,492 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=27, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-20 10:57:18,498 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded 2023-03-20 10:57:18,502 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A, FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B, FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C, FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/recovered.edits] 2023-03-20 10:57:18,506 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/67a2c2119ed44dbeb5005cf4283b052c to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/67a2c2119ed44dbeb5005cf4283b052c 2023-03-20 10:57:18,507 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/bee2e18c02964f8394b8ac02c527f2be to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/A/bee2e18c02964f8394b8ac02c527f2be 2023-03-20 10:57:18,509 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/6487b92b8c924dd48e0c3eb922919434 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/6487b92b8c924dd48e0c3eb922919434 2023-03-20 10:57:18,510 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/cf399f919455414db29d8bc1139304d0 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/B/cf399f919455414db29d8bc1139304d0 2023-03-20 10:57:18,512 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/53ed99f4dfdf4ce299806d594499a853 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/53ed99f4dfdf4ce299806d594499a853 2023-03-20 10:57:18,513 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/fe57225e4fb5439f8917f5ad61a0bdfb to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/C/fe57225e4fb5439f8917f5ad61a0bdfb 2023-03-20 10:57:18,515 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/recovered.edits/432.seqid to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded/recovered.edits/432.seqid 2023-03-20 10:57:18,516 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/9fc4d04a6b2183ce1217c4847d39aded 2023-03-20 10:57:18,517 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-20 10:57:18,517 DEBUG [PEWorker-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-20 10:57:18,518 DEBUG [PEWorker-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-03-20 10:57:18,521 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032003108f7fce2944bd843f30e339ba6f8d to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032003108f7fce2944bd843f30e339ba6f8d 2023-03-20 10:57:18,522 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032003d761d31d64421f9e76ac5710f35330 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032003d761d31d64421f9e76ac5710f35330 2023-03-20 10:57:18,524 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320057d4e48e2c74bd18d7b92cd9aaf0320 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320057d4e48e2c74bd18d7b92cd9aaf0320 2023-03-20 10:57:18,525 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320081f2df6916047549ed96eb56ce68474 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320081f2df6916047549ed96eb56ce68474 2023-03-20 10:57:18,525 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303201ebb2a4b260846c8b4fffb23a0fe7dd9 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303201ebb2a4b260846c8b4fffb23a0fe7dd9 2023-03-20 10:57:18,526 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320252afdc6d4644375ad3b00b9ed24306c to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320252afdc6d4644375ad3b00b9ed24306c 2023-03-20 10:57:18,527 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303202d21f4f58caf4a09866f45a99afba288 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303202d21f4f58caf4a09866f45a99afba288 2023-03-20 10:57:18,528 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320427640bc4d4643888e0bd1f2ae9e5f49 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320427640bc4d4643888e0bd1f2ae9e5f49 2023-03-20 10:57:18,529 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032043bd71e1a4b44ed7865482f7cc6a5864 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032043bd71e1a4b44ed7865482f7cc6a5864 2023-03-20 10:57:18,529 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303205136c2306a9048e08eb592be96eff8c4 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303205136c2306a9048e08eb592be96eff8c4 2023-03-20 10:57:18,530 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032058be4299b00e4f738fc6be540f737146 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032058be4299b00e4f738fc6be540f737146 2023-03-20 10:57:18,531 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303206438eb747679466683798447fd618194 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303206438eb747679466683798447fd618194 2023-03-20 10:57:18,532 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032069ddaf8d033c49389d26435eceb35940 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032069ddaf8d033c49389d26435eceb35940 2023-03-20 10:57:18,533 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303207c78023f78fc40399ad6718b469407a0 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303207c78023f78fc40399ad6718b469407a0 2023-03-20 10:57:18,534 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303207f4d9bca78a94b52a44dd97dc3a721f4 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303207f4d9bca78a94b52a44dd97dc3a721f4 2023-03-20 10:57:18,535 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303208164828578aa4e8fb2af26d8eb39a2b0 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303208164828578aa4e8fb2af26d8eb39a2b0 2023-03-20 10:57:18,536 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320994ab891137049a6bb9dc1929b24b518 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320994ab891137049a6bb9dc1929b24b518 2023-03-20 10:57:18,536 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320a04173d0920c43ad97938e352fc9cf4c to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320a04173d0920c43ad97938e352fc9cf4c 2023-03-20 10:57:18,538 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320af01a3bab4ee482283ba4cc17865bfa8 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320af01a3bab4ee482283ba4cc17865bfa8 2023-03-20 10:57:18,539 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320bb19f60cf2994be78cda41047bfe3a1a to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320bb19f60cf2994be78cda41047bfe3a1a 2023-03-20 10:57:18,540 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320c5219afa2269420496871b488fb6bc68 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320c5219afa2269420496871b488fb6bc68 2023-03-20 10:57:18,540 DEBUG [PEWorker-5] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-20 10:57:18,542 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=27, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-20 10:57:18,544 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-20 10:57:18,546 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-20 10:57:18,547 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=27, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-20 10:57:18,547 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-20 10:57:18,547 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1679309838547"}]},"ts":"9223372036854775807"} 2023-03-20 10:57:18,549 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-20 10:57:18,549 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 9fc4d04a6b2183ce1217c4847d39aded, NAME => 'TestAcidGuarantees,,1679309804748.9fc4d04a6b2183ce1217c4847d39aded.', STARTKEY => '', ENDKEY => ''}] 2023-03-20 10:57:18,549 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-20 10:57:18,550 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1679309838549"}]},"ts":"9223372036854775807"} 2023-03-20 10:57:18,552 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-20 10:57:18,553 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=27, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-20 10:57:18,555 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 66 msec 2023-03-20 10:57:18,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-03-20 10:57:18,594 INFO [Listener at localhost.localdomain/43509] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 27 completed 2023-03-20 10:57:18,613 INFO [Listener at localhost.localdomain/43509] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=230 (was 238), OpenFileDescriptor=545 (was 582), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=782 (was 709) - SystemLoadAverage LEAK? -, ProcessCount=184 (was 184), AvailableMemoryMB=8771 (was 9590) 2023-03-20 10:57:18,625 INFO [Listener at localhost.localdomain/43509] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=230, OpenFileDescriptor=545, MaxFileDescriptor=60000, SystemLoadAverage=782, ProcessCount=184, AvailableMemoryMB=8771 2023-03-20 10:57:18,628 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-20 10:57:18,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-20 10:57:18,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure2.ProcedureExecutor(1029): Stored pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-20 10:57:18,631 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-20 10:57:18,631 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 28 2023-03-20 10:57:18,632 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-20 10:57:18,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-20 10:57:18,635 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2 2023-03-20 10:57:18,636 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2 empty. 2023-03-20 10:57:18,636 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2 2023-03-20 10:57:18,636 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-20 10:57:18,643 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-20 10:57:18,646 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 72402f7d20e803d356627e208deb6fe2, NAME => 'TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp 2023-03-20 10:57:18,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-20 10:57:18,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-20 10:57:19,042 DEBUG [regionserver/jenkins-hbase20:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-03-20 10:57:19,052 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-20 10:57:19,052 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 72402f7d20e803d356627e208deb6fe2, disabling compactions & flushes 2023-03-20 10:57:19,052 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:19,052 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:19,052 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. after waiting 0 ms 2023-03-20 10:57:19,052 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:19,052 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:19,052 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 72402f7d20e803d356627e208deb6fe2: 2023-03-20 10:57:19,055 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-20 10:57:19,056 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679309839056"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679309839056"}]},"ts":"1679309839056"} 2023-03-20 10:57:19,058 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-20 10:57:19,058 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-20 10:57:19,059 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679309839058"}]},"ts":"1679309839058"} 2023-03-20 10:57:19,060 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-20 10:57:19,063 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=72402f7d20e803d356627e208deb6fe2, ASSIGN}] 2023-03-20 10:57:19,065 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=72402f7d20e803d356627e208deb6fe2, ASSIGN 2023-03-20 10:57:19,066 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=72402f7d20e803d356627e208deb6fe2, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,36041,1679309777062; forceNewPlan=false, retain=false 2023-03-20 10:57:19,217 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=72402f7d20e803d356627e208deb6fe2, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:19,217 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679309839217"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679309839217"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679309839217"}]},"ts":"1679309839217"} 2023-03-20 10:57:19,218 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; OpenRegionProcedure 72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062}] 2023-03-20 10:57:19,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-20 10:57:19,375 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:19,375 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 72402f7d20e803d356627e208deb6fe2, NAME => 'TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.', STARTKEY => '', ENDKEY => ''} 2023-03-20 10:57:19,375 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 72402f7d20e803d356627e208deb6fe2 2023-03-20 10:57:19,375 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-20 10:57:19,375 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 72402f7d20e803d356627e208deb6fe2 2023-03-20 10:57:19,376 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 72402f7d20e803d356627e208deb6fe2 2023-03-20 10:57:19,377 INFO [StoreOpener-72402f7d20e803d356627e208deb6fe2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 72402f7d20e803d356627e208deb6fe2 2023-03-20 10:57:19,379 DEBUG [StoreOpener-72402f7d20e803d356627e208deb6fe2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A 2023-03-20 10:57:19,379 DEBUG [StoreOpener-72402f7d20e803d356627e208deb6fe2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A 2023-03-20 10:57:19,379 INFO [StoreOpener-72402f7d20e803d356627e208deb6fe2-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-20 10:57:19,379 INFO [StoreOpener-72402f7d20e803d356627e208deb6fe2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 72402f7d20e803d356627e208deb6fe2 columnFamilyName A 2023-03-20 10:57:19,380 INFO [StoreOpener-72402f7d20e803d356627e208deb6fe2-1] regionserver.HStore(310): Store=72402f7d20e803d356627e208deb6fe2/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-20 10:57:19,380 INFO [StoreOpener-72402f7d20e803d356627e208deb6fe2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 72402f7d20e803d356627e208deb6fe2 2023-03-20 10:57:19,381 DEBUG [StoreOpener-72402f7d20e803d356627e208deb6fe2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B 2023-03-20 10:57:19,381 DEBUG [StoreOpener-72402f7d20e803d356627e208deb6fe2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B 2023-03-20 10:57:19,381 INFO [StoreOpener-72402f7d20e803d356627e208deb6fe2-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-20 10:57:19,381 INFO [StoreOpener-72402f7d20e803d356627e208deb6fe2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 72402f7d20e803d356627e208deb6fe2 columnFamilyName B 2023-03-20 10:57:19,382 INFO [StoreOpener-72402f7d20e803d356627e208deb6fe2-1] regionserver.HStore(310): Store=72402f7d20e803d356627e208deb6fe2/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-20 10:57:19,382 INFO [StoreOpener-72402f7d20e803d356627e208deb6fe2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 72402f7d20e803d356627e208deb6fe2 2023-03-20 10:57:19,383 DEBUG [StoreOpener-72402f7d20e803d356627e208deb6fe2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C 2023-03-20 10:57:19,383 DEBUG [StoreOpener-72402f7d20e803d356627e208deb6fe2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C 2023-03-20 10:57:19,383 INFO [StoreOpener-72402f7d20e803d356627e208deb6fe2-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-20 10:57:19,383 INFO [StoreOpener-72402f7d20e803d356627e208deb6fe2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 72402f7d20e803d356627e208deb6fe2 columnFamilyName C 2023-03-20 10:57:19,384 INFO [StoreOpener-72402f7d20e803d356627e208deb6fe2-1] regionserver.HStore(310): Store=72402f7d20e803d356627e208deb6fe2/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-20 10:57:19,384 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:19,385 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2 2023-03-20 10:57:19,385 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2 2023-03-20 10:57:19,387 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-20 10:57:19,389 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 72402f7d20e803d356627e208deb6fe2 2023-03-20 10:57:19,391 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-20 10:57:19,392 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 72402f7d20e803d356627e208deb6fe2; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65556070, jitterRate=-0.023138433694839478}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-20 10:57:19,392 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 72402f7d20e803d356627e208deb6fe2: 2023-03-20 10:57:19,393 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2., pid=30, masterSystemTime=1679309839369 2023-03-20 10:57:19,394 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:19,394 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:19,394 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=72402f7d20e803d356627e208deb6fe2, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:19,395 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679309839394"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1679309839394"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679309839394"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679309839394"}]},"ts":"1679309839394"} 2023-03-20 10:57:19,397 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-03-20 10:57:19,397 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; OpenRegionProcedure 72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 in 178 msec 2023-03-20 10:57:19,399 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-03-20 10:57:19,399 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=72402f7d20e803d356627e208deb6fe2, ASSIGN in 335 msec 2023-03-20 10:57:19,399 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-20 10:57:19,399 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679309839399"}]},"ts":"1679309839399"} 2023-03-20 10:57:19,400 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-20 10:57:19,402 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-20 10:57:19,403 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=28, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 774 msec 2023-03-20 10:57:19,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-20 10:57:19,737 INFO [Listener at localhost.localdomain/43509] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 28 completed 2023-03-20 10:57:19,737 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(139): Connect 0x4d3f4b11 to 127.0.0.1:60226 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 10:57:19,742 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5c466b6c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 10:57:19,743 DEBUG [Listener at localhost.localdomain/43509] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 10:57:19,745 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:55100, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 10:57:19,746 DEBUG [Listener at localhost.localdomain/43509] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-20 10:57:19,748 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:49740, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-20 10:57:19,749 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(139): Connect 0x7187fcf3 to 127.0.0.1:60226 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 10:57:19,754 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@46d74611, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 10:57:19,754 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(139): Connect 0x712d78ca to 127.0.0.1:60226 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 10:57:19,758 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3f837278, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 10:57:19,758 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(139): Connect 0x407b9cb0 to 127.0.0.1:60226 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 10:57:19,762 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5a8a575d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 10:57:19,762 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(139): Connect 0x7f456fa2 to 127.0.0.1:60226 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 10:57:19,771 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6347f2e6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 10:57:19,772 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(139): Connect 0x62be7a0b to 127.0.0.1:60226 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 10:57:19,780 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@36523181, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 10:57:19,780 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(139): Connect 0x6b458e37 to 127.0.0.1:60226 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 10:57:19,789 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4cbc74e7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 10:57:19,790 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(139): Connect 0x7bae1a64 to 127.0.0.1:60226 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 10:57:19,804 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3c5d1710, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 10:57:19,805 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(139): Connect 0x5654f8ca to 127.0.0.1:60226 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 10:57:19,812 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5d22a55e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 10:57:19,813 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(139): Connect 0x509adaa8 to 127.0.0.1:60226 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 10:57:19,821 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2179ead9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 10:57:19,821 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(139): Connect 0x7e5e7a93 to 127.0.0.1:60226 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 10:57:19,829 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7897b06a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 10:57:19,836 DEBUG [hconnection-0x503b0126-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 10:57:19,836 DEBUG [hconnection-0xdab5940-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 10:57:19,836 DEBUG [hconnection-0x10481cb1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 10:57:19,838 DEBUG [hconnection-0x9723333-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 10:57:19,838 DEBUG [hconnection-0xf094dee-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 10:57:19,839 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:55132, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 10:57:19,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:19,844 DEBUG [hconnection-0x477a9640-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 10:57:19,846 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:55150, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 10:57:19,850 DEBUG [hconnection-0x4af41344-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 10:57:19,875 DEBUG [hconnection-0x2667bdb3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 10:57:19,877 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:55118, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 10:57:19,877 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:55116, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 10:57:19,878 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:55144, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 10:57:19,878 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:55136, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 10:57:19,878 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:55158, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 10:57:19,878 DEBUG [hconnection-0x341f394d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 10:57:19,883 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:55168, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 10:57:19,893 DEBUG [hconnection-0x6f590bb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 10:57:19,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:19,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:19,896 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:19,896 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:19,896 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:19,896 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:19,897 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:19,897 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:19,898 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:19,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:19,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:19,898 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:19,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:19,899 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:19,899 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:19,899 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:19,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:19,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:19,899 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:19,905 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:19,905 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:19,905 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:19,905 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:19,905 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:19,905 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:19,905 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:19,906 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:19,906 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. started... 2023-03-20 10:57:19,910 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 72402f7d20e803d356627e208deb6fe2 3/3 column families, dataSize=6.71 KB heapSize=20.67 KB 2023-03-20 10:57:19,911 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:55182, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 10:57:19,911 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:55160, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 10:57:19,922 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=A 2023-03-20 10:57:19,922 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:19,922 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=B 2023-03-20 10:57:19,922 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:19,922 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=C 2023-03-20 10:57:19,922 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:19,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 72402f7d20e803d356627e208deb6fe2 2023-03-20 10:57:19,928 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. as already flushing 2023-03-20 10:57:19,955 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679309839942, max timestamp=1679309839953Num uniques 859; 2023-03-20 10:57:19,955 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-20 10:57:19,955 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679309839942, max timestamp=1679309839953Num uniques 859; 2023-03-20 10:57:19,955 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-20 10:57:19,955 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679309839937, max timestamp=1679309839953Num uniques 859; 2023-03-20 10:57:19,955 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-20 10:57:19,979 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679309839970, max timestamp=1679309839975Num uniques 1018; 2023-03-20 10:57:19,979 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679309839970, max timestamp=1679309839975Num uniques 1018; 2023-03-20 10:57:19,979 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-20 10:57:19,979 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679309839970, max timestamp=1679309839975Num uniques 1118; 2023-03-20 10:57:19,979 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-20 10:57:19,979 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=9978, suffix heap size=144240,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=218 2023-03-20 10:57:20,000 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679309839993, max timestamp=1679309839997Num uniques 1077; 2023-03-20 10:57:20,000 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-03-20 10:57:20,000 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679309839993, max timestamp=1679309839997Num uniques 1077; 2023-03-20 10:57:20,001 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-03-20 10:57:20,002 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679309839993, max timestamp=1679309839996Num uniques 1077; 2023-03-20 10:57:20,002 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-03-20 10:57:20,024 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1679309840013, max timestamp=1679309840020Num uniques 1036; 2023-03-20 10:57:20,024 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-03-20 10:57:20,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1679309840013, max timestamp=1679309840020Num uniques 1036; 2023-03-20 10:57:20,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-03-20 10:57:20,026 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679309840013, max timestamp=1679309840020Num uniques 1036; 2023-03-20 10:57:20,026 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=6870, suffix heap size=135712,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=150 2023-03-20 10:57:20,056 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1679309840048, max timestamp=1679309840053Num uniques 1095; 2023-03-20 10:57:20,056 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-03-20 10:57:20,059 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1679309840048, max timestamp=1679309840053Num uniques 1009; 2023-03-20 10:57:20,059 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1679309840048, max timestamp=1679309840053Num uniques 1095; 2023-03-20 10:57:20,059 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8930, suffix heap size=132904,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=195 2023-03-20 10:57:20,059 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-03-20 10:57:20,083 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11624, totalHeapSize=26656, min timestamp=1679309840071, max timestamp=1679309840081Num uniques 1104; 2023-03-20 10:57:20,084 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=50554, new segment data size=11624, suffix heap size=142784,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=254 2023-03-20 10:57:20,086 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679309840073, max timestamp=1679309840082Num uniques 1054; 2023-03-20 10:57:20,086 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679309840073, max timestamp=1679309840082Num uniques 1054; 2023-03-20 10:57:20,086 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-03-20 10:57:20,086 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-03-20 10:57:20,110 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1679309840102, max timestamp=1679309840107Num uniques 1013; 2023-03-20 10:57:20,110 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-03-20 10:57:20,110 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1679309840102, max timestamp=1679309840107Num uniques 1113; 2023-03-20 10:57:20,110 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=9748, suffix heap size=143720,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=213 2023-03-20 10:57:20,112 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1679309840102, max timestamp=1679309840107Num uniques 1013; 2023-03-20 10:57:20,112 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-03-20 10:57:20,136 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1679309840124, max timestamp=1679309840133Num uniques 1072; 2023-03-20 10:57:20,136 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1679309840124, max timestamp=1679309840133Num uniques 1022; 2023-03-20 10:57:20,136 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-03-20 10:57:20,136 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=10162, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-03-20 10:57:20,149 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679309840124, max timestamp=1679309840133Num uniques 1022; 2023-03-20 10:57:20,149 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=6870, suffix heap size=134256,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=150 2023-03-20 10:57:20,161 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1679309840156, max timestamp=1679309840158Num uniques 1081; 2023-03-20 10:57:20,161 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=8286, suffix heap size=140392,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=181 2023-03-20 10:57:20,161 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1679309840156, max timestamp=1679309840158Num uniques 1081; 2023-03-20 10:57:20,161 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=8286, suffix heap size=140392,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=181 2023-03-20 10:57:20,163 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1679309840156, max timestamp=1679309840158Num uniques 1009; 2023-03-20 10:57:20,163 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8286, suffix heap size=132904,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=181 2023-03-20 10:57:20,188 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679309840175, max timestamp=1679309840185Num uniques 1040; 2023-03-20 10:57:20,188 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=47620, new segment data size=6870, suffix heap size=136128,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=150 2023-03-20 10:57:20,190 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679309840175, max timestamp=1679309840185Num uniques 1040; 2023-03-20 10:57:20,190 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=47624, new segment data size=6870, suffix heap size=136128,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=150 2023-03-20 10:57:20,193 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679309840175, max timestamp=1679309840185Num uniques 1040; 2023-03-20 10:57:20,193 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=47623, new segment data size=6870, suffix heap size=136128,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=150 2023-03-20 10:57:20,212 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1679309840200, max timestamp=1679309840209Num uniques 1009; 2023-03-20 10:57:20,212 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1679309840200, max timestamp=1679309840209Num uniques 1009; 2023-03-20 10:57:20,212 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=46221, new segment data size=9114, suffix heap size=132904,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=199 2023-03-20 10:57:20,212 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=46220, new segment data size=9114, suffix heap size=132904,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=199 2023-03-20 10:57:20,213 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1679309840200, max timestamp=1679309840209Num uniques 1009; 2023-03-20 10:57:20,213 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=46224, new segment data size=9114, suffix heap size=132904,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=199 2023-03-20 10:57:20,238 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1679309840227, max timestamp=1679309840235Num uniques 1058; 2023-03-20 10:57:20,238 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=7230, suffix heap size=138000,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=158 2023-03-20 10:57:20,239 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679309840227, max timestamp=1679309840235Num uniques 1058; 2023-03-20 10:57:20,239 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=6870, suffix heap size=138000,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=150 2023-03-20 10:57:20,240 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1679309840227, max timestamp=1679309840235Num uniques 1058; 2023-03-20 10:57:20,240 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=7230, suffix heap size=138000,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=158 2023-03-20 10:57:20,256 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=267, cellSize=12222, totalHeapSize=28008, min timestamp=1679309840232, max timestamp=1679309840254Num uniques 1017; 2023-03-20 10:57:20,256 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=12222, suffix heap size=133736,new segment heap size=28008 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=267 2023-03-20 10:57:20,257 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1679309840251, max timestamp=1679309840254Num uniques 1017; 2023-03-20 10:57:20,257 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=7642, suffix heap size=133736,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=167 2023-03-20 10:57:20,260 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1679309840251, max timestamp=1679309840256Num uniques 1009; 2023-03-20 10:57:20,260 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=46212, new segment data size=7642, suffix heap size=132904,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=167 2023-03-20 10:57:20,290 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1679309840281, max timestamp=1679309840288Num uniques 1026; 2023-03-20 10:57:20,291 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=8056, suffix heap size=134672,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=176 2023-03-20 10:57:20,291 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1679309840281, max timestamp=1679309840288Num uniques 1126; 2023-03-20 10:57:20,291 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=51566, new segment data size=8056, suffix heap size=145072,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1126, new segment cells count=176 2023-03-20 10:57:20,292 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679309840281, max timestamp=1679309840288Num uniques 1026; 2023-03-20 10:57:20,292 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=6870, suffix heap size=134672,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=150 2023-03-20 10:57:20,310 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1679309840296, max timestamp=1679309840306Num uniques 1035; 2023-03-20 10:57:20,310 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=10760, suffix heap size=135608,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=235 2023-03-20 10:57:20,310 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1679309840296, max timestamp=1679309840306Num uniques 1035; 2023-03-20 10:57:20,310 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=10760, suffix heap size=135608,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=235 2023-03-20 10:57:20,312 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679309840296, max timestamp=1679309840306Num uniques 1009; 2023-03-20 10:57:20,312 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-03-20 10:57:20,335 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1679309840320, max timestamp=1679309840328Num uniques 1094; 2023-03-20 10:57:20,336 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=11174, suffix heap size=141744,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=244 2023-03-20 10:57:20,336 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1679309840320, max timestamp=1679309840331Num uniques 1094; 2023-03-20 10:57:20,336 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=8884, suffix heap size=141744,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=194 2023-03-20 10:57:20,336 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1679309840320, max timestamp=1679309840331Num uniques 1009; 2023-03-20 10:57:20,336 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8884, suffix heap size=132904,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=194 2023-03-20 10:57:20,360 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=7 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/A/e0f2074420494f18892dd06834b239df 2023-03-20 10:57:20,366 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1679309840354, max timestamp=1679309840361Num uniques 1103; 2023-03-20 10:57:20,366 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=50515, new segment data size=7005, suffix heap size=142680,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=153 2023-03-20 10:57:20,367 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1679309840354, max timestamp=1679309840361Num uniques 1053; 2023-03-20 10:57:20,368 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=7005, suffix heap size=137480,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=153 2023-03-20 10:57:20,370 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1679309840354, max timestamp=1679309840361Num uniques 1053; 2023-03-20 10:57:20,370 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=7005, suffix heap size=137480,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=153 2023-03-20 10:57:20,396 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=212, cellSize=9702, totalHeapSize=22288, min timestamp=1679309840386, max timestamp=1679309840394Num uniques 1012; 2023-03-20 10:57:20,396 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=9702, suffix heap size=133216,new segment heap size=22288 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=212 2023-03-20 10:57:20,397 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=212, cellSize=9702, totalHeapSize=22288, min timestamp=1679309840386, max timestamp=1679309840394Num uniques 1012; 2023-03-20 10:57:20,397 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=9702, suffix heap size=133216,new segment heap size=22288 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=212 2023-03-20 10:57:20,398 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=212, cellSize=9702, totalHeapSize=22288, min timestamp=1679309840386, max timestamp=1679309840394Num uniques 1012; 2023-03-20 10:57:20,398 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=9702, suffix heap size=133216,new segment heap size=22288 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=212 2023-03-20 10:57:20,464 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7826, totalHeapSize=18024, min timestamp=1679309840403, max timestamp=1679309840462Num uniques 1071; 2023-03-20 10:57:20,464 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=49046, new segment data size=7826, suffix heap size=139352,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1071, new segment cells count=171 2023-03-20 10:57:20,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7826, totalHeapSize=18024, min timestamp=1679309840403, max timestamp=1679309840462Num uniques 1071; 2023-03-20 10:57:20,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=49046, new segment data size=7826, suffix heap size=139352,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1071, new segment cells count=171 2023-03-20 10:57:20,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7826, totalHeapSize=18024, min timestamp=1679309840403, max timestamp=1679309840462Num uniques 1071; 2023-03-20 10:57:20,465 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=49046, new segment data size=7826, suffix heap size=139352,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1071, new segment cells count=171 2023-03-20 10:57:20,498 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=180, cellSize=8240, totalHeapSize=18960, min timestamp=1679309840481, max timestamp=1679309840489Num uniques 1030; 2023-03-20 10:57:20,498 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=47170, new segment data size=8240, suffix heap size=135088,new segment heap size=18960 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1030, new segment cells count=180 2023-03-20 10:57:20,499 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=180, cellSize=8240, totalHeapSize=18960, min timestamp=1679309840481, max timestamp=1679309840489Num uniques 1030; 2023-03-20 10:57:20,499 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=47170, new segment data size=8240, suffix heap size=135088,new segment heap size=18960 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1030, new segment cells count=180 2023-03-20 10:57:20,499 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=180, cellSize=8240, totalHeapSize=18960, min timestamp=1679309840481, max timestamp=1679309840489Num uniques 1030; 2023-03-20 10:57:20,499 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=47170, new segment data size=8240, suffix heap size=135088,new segment heap size=18960 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1030, new segment cells count=180 2023-03-20 10:57:20,527 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=189, cellSize=8654, totalHeapSize=19896, min timestamp=1679309840514, max timestamp=1679309840525Num uniques 1039; 2023-03-20 10:57:20,527 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=189, cellSize=8654, totalHeapSize=19896, min timestamp=1679309840514, max timestamp=1679309840525Num uniques 1039; 2023-03-20 10:57:20,527 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=47584, new segment data size=8654, suffix heap size=136024,new segment heap size=19896 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1039, new segment cells count=189 2023-03-20 10:57:20,527 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=189, cellSize=8654, totalHeapSize=19896, min timestamp=1679309840514, max timestamp=1679309840525Num uniques 1039; 2023-03-20 10:57:20,527 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=47584, new segment data size=8654, suffix heap size=136024,new segment heap size=19896 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1039, new segment cells count=189 2023-03-20 10:57:20,527 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=47584, new segment data size=8654, suffix heap size=136024,new segment heap size=19896 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1039, new segment cells count=189 2023-03-20 10:57:20,541 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=248, cellSize=11358, totalHeapSize=26032, min timestamp=1679309840533, max timestamp=1679309840539Num uniques 1048; 2023-03-20 10:57:20,542 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=47998, new segment data size=11358, suffix heap size=136960,new segment heap size=26032 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1048, new segment cells count=248 2023-03-20 10:57:20,542 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=248, cellSize=11358, totalHeapSize=26032, min timestamp=1679309840533, max timestamp=1679309840539Num uniques 1048; 2023-03-20 10:57:20,542 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=47998, new segment data size=11358, suffix heap size=136960,new segment heap size=26032 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1048, new segment cells count=248 2023-03-20 10:57:20,542 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=248, cellSize=11358, totalHeapSize=26032, min timestamp=1679309840533, max timestamp=1679309840539Num uniques 1048; 2023-03-20 10:57:20,542 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=47998, new segment data size=11358, suffix heap size=136960,new segment heap size=26032 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1048, new segment cells count=248 2023-03-20 10:57:20,562 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=207, cellSize=9475, totalHeapSize=21768, min timestamp=1679309840555, max timestamp=1679309840559Num uniques 1107; 2023-03-20 10:57:20,562 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=207, cellSize=9475, totalHeapSize=21768, min timestamp=1679309840555, max timestamp=1679309840559Num uniques 1107; 2023-03-20 10:57:20,562 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679309840555, max timestamp=1679309840559Num uniques 1107; 2023-03-20 10:57:20,562 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=50695, new segment data size=9475, suffix heap size=143096,new segment heap size=21768 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1107, new segment cells count=207 2023-03-20 10:57:20,562 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=50695, new segment data size=9475, suffix heap size=143096,new segment heap size=21768 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1107, new segment cells count=207 2023-03-20 10:57:20,562 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=50695, new segment data size=6870, suffix heap size=143096,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1107, new segment cells count=150 2023-03-20 10:57:20,576 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=266, cellSize=12176, totalHeapSize=27904, min timestamp=1679309840568, max timestamp=1679309840574Num uniques 1066; 2023-03-20 10:57:20,576 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=266, cellSize=12176, totalHeapSize=27904, min timestamp=1679309840568, max timestamp=1679309840574Num uniques 1066; 2023-03-20 10:57:20,576 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=48816, new segment data size=12176, suffix heap size=138832,new segment heap size=27904 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1066, new segment cells count=266 2023-03-20 10:57:20,576 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=48816, new segment data size=12176, suffix heap size=138832,new segment heap size=27904 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1066, new segment cells count=266 2023-03-20 10:57:20,576 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=266, cellSize=12176, totalHeapSize=27904, min timestamp=1679309840568, max timestamp=1679309840574Num uniques 1009; 2023-03-20 10:57:20,577 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=46211, new segment data size=12176, suffix heap size=132904,new segment heap size=27904 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=266 2023-03-20 10:57:20,591 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=225, cellSize=10300, totalHeapSize=23640, min timestamp=1679309840582, max timestamp=1679309840589Num uniques 1125; 2023-03-20 10:57:20,591 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=51520, new segment data size=10300, suffix heap size=144968,new segment heap size=23640 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1125, new segment cells count=225 2023-03-20 10:57:20,591 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=225, cellSize=10300, totalHeapSize=23640, min timestamp=1679309840582, max timestamp=1679309840589Num uniques 1125; 2023-03-20 10:57:20,591 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=225, cellSize=10300, totalHeapSize=23640, min timestamp=1679309840582, max timestamp=1679309840589Num uniques 1125; 2023-03-20 10:57:20,591 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=51520, new segment data size=10300, suffix heap size=144968,new segment heap size=23640 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1125, new segment cells count=225 2023-03-20 10:57:20,591 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=51520, new segment data size=10300, suffix heap size=144968,new segment heap size=23640 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1125, new segment cells count=225 2023-03-20 10:57:20,621 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679309840600, max timestamp=1679309840613Num uniques 1084; 2023-03-20 10:57:20,621 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=49634, new segment data size=6870, suffix heap size=140704,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1084, new segment cells count=150 2023-03-20 10:57:20,622 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679309840602, max timestamp=1679309840613Num uniques 1084; 2023-03-20 10:57:20,622 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=49634, new segment data size=6870, suffix heap size=140704,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1084, new segment cells count=150 2023-03-20 10:57:20,624 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679309840600, max timestamp=1679309840613Num uniques 1084; 2023-03-20 10:57:20,626 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=49634, new segment data size=6870, suffix heap size=140704,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1084, new segment cells count=150 2023-03-20 10:57:20,669 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:20,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55182 deadline: 1679309900666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:20,670 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:20,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55136 deadline: 1679309900667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:20,671 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:20,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55168 deadline: 1679309900668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:20,671 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:20,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55132 deadline: 1679309900668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:20,672 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679309840643, max timestamp=1679309840665Num uniques 1009; 2023-03-20 10:57:20,672 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:20,672 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=46224, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-03-20 10:57:20,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55118 deadline: 1679309900669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:20,674 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679309840643, max timestamp=1679309840665Num uniques 1009; 2023-03-20 10:57:20,674 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=46224, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-03-20 10:57:20,676 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679309840643, max timestamp=1679309840665Num uniques 1009; 2023-03-20 10:57:20,676 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=46224, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-03-20 10:57:20,787 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=152, cellSize=6960, totalHeapSize=16048, min timestamp=1679309840781, max timestamp=1679309840785Num uniques 1009; 2023-03-20 10:57:20,788 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=152, cellSize=6960, totalHeapSize=16048, min timestamp=1679309840781, max timestamp=1679309840785Num uniques 1009; 2023-03-20 10:57:20,788 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=46212, new segment data size=6960, suffix heap size=132904,new segment heap size=16048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=152 2023-03-20 10:57:20,788 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=46212, new segment data size=6960, suffix heap size=132904,new segment heap size=16048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=152 2023-03-20 10:57:20,789 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679309840781, max timestamp=1679309840785Num uniques 1009; 2023-03-20 10:57:20,790 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=46212, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-03-20 10:57:20,800 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=7 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/B/fc0188b7d1fa415fb217910da8214522 2023-03-20 10:57:20,803 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=211, cellSize=9655, totalHeapSize=22184, min timestamp=1679309840793, max timestamp=1679309840801Num uniques 1011; 2023-03-20 10:57:20,804 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=46295, new segment data size=9655, suffix heap size=133112,new segment heap size=22184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1011, new segment cells count=211 2023-03-20 10:57:20,804 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=211, cellSize=9655, totalHeapSize=22184, min timestamp=1679309840793, max timestamp=1679309840801Num uniques 1011; 2023-03-20 10:57:20,804 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=46295, new segment data size=9655, suffix heap size=133112,new segment heap size=22184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1011, new segment cells count=211 2023-03-20 10:57:20,804 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=211, cellSize=9656, totalHeapSize=22184, min timestamp=1679309840793, max timestamp=1679309840801Num uniques 1009; 2023-03-20 10:57:20,804 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=46206, new segment data size=9656, suffix heap size=132904,new segment heap size=22184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=211 2023-03-20 10:57:20,823 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=220, cellSize=10070, totalHeapSize=23120, min timestamp=1679309840812, max timestamp=1679309840821Num uniques 1070; 2023-03-20 10:57:20,823 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=49000, new segment data size=10070, suffix heap size=139248,new segment heap size=23120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1070, new segment cells count=220 2023-03-20 10:57:20,823 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=220, cellSize=10070, totalHeapSize=23120, min timestamp=1679309840812, max timestamp=1679309840821Num uniques 1070; 2023-03-20 10:57:20,823 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=49000, new segment data size=10070, suffix heap size=139248,new segment heap size=23120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1070, new segment cells count=220 2023-03-20 10:57:20,823 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=220, cellSize=10070, totalHeapSize=23120, min timestamp=1679309840812, max timestamp=1679309840821Num uniques 1070; 2023-03-20 10:57:20,823 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=49000, new segment data size=10070, suffix heap size=139248,new segment heap size=23120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1070, new segment cells count=220 2023-03-20 10:57:20,832 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=229, cellSize=10484, totalHeapSize=24056, min timestamp=1679309840828, max timestamp=1679309840830Num uniques 1079; 2023-03-20 10:57:20,832 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=229, cellSize=10484, totalHeapSize=24056, min timestamp=1679309840828, max timestamp=1679309840830Num uniques 1079; 2023-03-20 10:57:20,832 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=49414, new segment data size=10484, suffix heap size=140184,new segment heap size=24056 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1079, new segment cells count=229 2023-03-20 10:57:20,832 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=49414, new segment data size=10484, suffix heap size=140184,new segment heap size=24056 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1079, new segment cells count=229 2023-03-20 10:57:20,833 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=229, cellSize=10484, totalHeapSize=24056, min timestamp=1679309840828, max timestamp=1679309840830Num uniques 1079; 2023-03-20 10:57:20,833 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=49414, new segment data size=10484, suffix heap size=140184,new segment heap size=24056 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1079, new segment cells count=229 2023-03-20 10:57:20,845 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:20,845 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:20,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55136 deadline: 1679309900842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:20,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55118 deadline: 1679309900843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:20,846 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:20,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55182 deadline: 1679309900844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:20,846 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:20,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:55132 deadline: 1679309900845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:20,847 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679309840840, max timestamp=1679309840842Num uniques 1088; 2023-03-20 10:57:20,847 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=49828, new segment data size=6870, suffix heap size=141120,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1088, new segment cells count=150 2023-03-20 10:57:20,848 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679309840840, max timestamp=1679309840842Num uniques 1088; 2023-03-20 10:57:20,848 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=49828, new segment data size=6870, suffix heap size=141120,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1088, new segment cells count=150 2023-03-20 10:57:20,848 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679309840840, max timestamp=1679309840842Num uniques 1088; 2023-03-20 10:57:20,848 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=49828, new segment data size=6870, suffix heap size=141120,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1088, new segment cells count=150 2023-03-20 10:57:20,885 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=197, cellSize=9022, totalHeapSize=20728, min timestamp=1679309840870, max timestamp=1679309840883Num uniques 1009; 2023-03-20 10:57:20,885 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=197, cellSize=9022, totalHeapSize=20728, min timestamp=1679309840870, max timestamp=1679309840883Num uniques 1009; 2023-03-20 10:57:20,885 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=9022, suffix heap size=132904,new segment heap size=20728 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=197 2023-03-20 10:57:20,885 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=9022, suffix heap size=132904,new segment heap size=20728 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=197 2023-03-20 10:57:20,886 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=197, cellSize=9022, totalHeapSize=20728, min timestamp=1679309840870, max timestamp=1679309840883Num uniques 1009; 2023-03-20 10:57:20,886 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=9022, suffix heap size=132904,new segment heap size=20728 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=197 2023-03-20 10:57:20,944 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=156, cellSize=7140, totalHeapSize=16464, min timestamp=1679309840907, max timestamp=1679309840940Num uniques 1056; 2023-03-20 10:57:20,944 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=156, cellSize=7140, totalHeapSize=16464, min timestamp=1679309840907, max timestamp=1679309840940Num uniques 1056; 2023-03-20 10:57:20,944 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=156, cellSize=7140, totalHeapSize=16464, min timestamp=1679309840907, max timestamp=1679309840940Num uniques 1056; 2023-03-20 10:57:20,944 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=48360, new segment data size=7140, suffix heap size=137792,new segment heap size=16464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1056, new segment cells count=156 2023-03-20 10:57:20,944 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=48360, new segment data size=7140, suffix heap size=137792,new segment heap size=16464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1056, new segment cells count=156 2023-03-20 10:57:20,944 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=48360, new segment data size=7140, suffix heap size=137792,new segment heap size=16464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1056, new segment cells count=156 2023-03-20 10:57:20,975 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=165, cellSize=7545, totalHeapSize=17400, min timestamp=1679309840966, max timestamp=1679309840972Num uniques 1015; 2023-03-20 10:57:20,975 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=165, cellSize=7545, totalHeapSize=17400, min timestamp=1679309840966, max timestamp=1679309840972Num uniques 1015; 2023-03-20 10:57:20,975 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=46475, new segment data size=7545, suffix heap size=133528,new segment heap size=17400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1015, new segment cells count=165 2023-03-20 10:57:20,975 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=46475, new segment data size=7545, suffix heap size=133528,new segment heap size=17400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1015, new segment cells count=165 2023-03-20 10:57:20,975 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=165, cellSize=7545, totalHeapSize=17400, min timestamp=1679309840966, max timestamp=1679309840972Num uniques 1015; 2023-03-20 10:57:20,976 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=46475, new segment data size=7545, suffix heap size=133528,new segment heap size=17400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1015, new segment cells count=165 2023-03-20 10:57:20,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=274, cellSize=12535, totalHeapSize=28736, min timestamp=1679309840985, max timestamp=1679309840995Num uniques 1024; 2023-03-20 10:57:20,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=274, cellSize=12544, totalHeapSize=28736, min timestamp=1679309840985, max timestamp=1679309840995Num uniques 1024; 2023-03-20 10:57:20,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=46885, new segment data size=12535, suffix heap size=134464,new segment heap size=28736 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1024, new segment cells count=274 2023-03-20 10:57:20,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=46894, new segment data size=12544, suffix heap size=134464,new segment heap size=28736 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1024, new segment cells count=274 2023-03-20 10:57:20,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=274, cellSize=12544, totalHeapSize=28736, min timestamp=1679309840985, max timestamp=1679309840995Num uniques 1024; 2023-03-20 10:57:20,998 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=46894, new segment data size=12544, suffix heap size=134464,new segment heap size=28736 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1024, new segment cells count=274 2023-03-20 10:57:21,010 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=383, cellSize=17538, totalHeapSize=40072, min timestamp=1679309841001, max timestamp=1679309841007Num uniques 1133; 2023-03-20 10:57:21,010 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=51888, new segment data size=17538, suffix heap size=145800,new segment heap size=40072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1133, new segment cells count=383 2023-03-20 10:57:21,010 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=183, cellSize=8378, totalHeapSize=19272, min timestamp=1679309841005, max timestamp=1679309841007Num uniques 1133; 2023-03-20 10:57:21,010 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=51888, new segment data size=8378, suffix heap size=145800,new segment heap size=19272 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1133, new segment cells count=183 2023-03-20 10:57:21,010 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=383, cellSize=17538, totalHeapSize=40072, min timestamp=1679309841001, max timestamp=1679309841007Num uniques 1133; 2023-03-20 10:57:21,010 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=51888, new segment data size=17538, suffix heap size=145800,new segment heap size=40072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1133, new segment cells count=383 2023-03-20 10:57:21,027 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=192, cellSize=8792, totalHeapSize=20208, min timestamp=1679309841019, max timestamp=1679309841024Num uniques 1242; 2023-03-20 10:57:21,027 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=56882, new segment data size=8792, suffix heap size=157136,new segment heap size=20208 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1242, new segment cells count=192 2023-03-20 10:57:21,027 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=192, cellSize=8792, totalHeapSize=20208, min timestamp=1679309841019, max timestamp=1679309841024Num uniques 1242; 2023-03-20 10:57:21,027 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=56882, new segment data size=8792, suffix heap size=157136,new segment heap size=20208 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1242, new segment cells count=192 2023-03-20 10:57:21,027 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=152, cellSize=6960, totalHeapSize=16048, min timestamp=1679309841019, max timestamp=1679309841024Num uniques 1042; 2023-03-20 10:57:21,028 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=47720, new segment data size=6960, suffix heap size=136336,new segment heap size=16048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1042, new segment cells count=152 2023-03-20 10:57:21,049 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=151, cellSize=6915, totalHeapSize=15944, min timestamp=1679309841034, max timestamp=1679309841046Num uniques 1011; 2023-03-20 10:57:21,049 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=46305, new segment data size=6915, suffix heap size=133112,new segment heap size=15944 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1011, new segment cells count=151 2023-03-20 10:57:21,050 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=151, cellSize=6915, totalHeapSize=15944, min timestamp=1679309841034, max timestamp=1679309841046Num uniques 1051; 2023-03-20 10:57:21,050 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=48135, new segment data size=6915, suffix heap size=137272,new segment heap size=15944 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1051, new segment cells count=151 2023-03-20 10:57:21,053 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=151, cellSize=6915, totalHeapSize=15944, min timestamp=1679309841034, max timestamp=1679309841046Num uniques 1051; 2023-03-20 10:57:21,053 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=48135, new segment data size=6915, suffix heap size=137272,new segment heap size=15944 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1051, new segment cells count=151 2023-03-20 10:57:21,070 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=210, cellSize=9610, totalHeapSize=22080, min timestamp=1679309841059, max timestamp=1679309841068Num uniques 1010; 2023-03-20 10:57:21,070 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=46250, new segment data size=9610, suffix heap size=133008,new segment heap size=22080 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1010, new segment cells count=210 2023-03-20 10:57:21,070 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=210, cellSize=9610, totalHeapSize=22080, min timestamp=1679309841059, max timestamp=1679309841068Num uniques 1010; 2023-03-20 10:57:21,070 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=46250, new segment data size=9610, suffix heap size=133008,new segment heap size=22080 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1010, new segment cells count=210 2023-03-20 10:57:21,070 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=210, cellSize=9610, totalHeapSize=22080, min timestamp=1679309841059, max timestamp=1679309841068Num uniques 1010; 2023-03-20 10:57:21,070 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=46250, new segment data size=9610, suffix heap size=133008,new segment heap size=22080 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1010, new segment cells count=210 2023-03-20 10:57:21,082 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=219, cellSize=10024, totalHeapSize=23016, min timestamp=1679309841075, max timestamp=1679309841079Num uniques 1069; 2023-03-20 10:57:21,082 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=48954, new segment data size=10024, suffix heap size=139144,new segment heap size=23016 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1069, new segment cells count=219 2023-03-20 10:57:21,082 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=219, cellSize=10024, totalHeapSize=23016, min timestamp=1679309841075, max timestamp=1679309841079Num uniques 1069; 2023-03-20 10:57:21,082 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=219, cellSize=10024, totalHeapSize=23016, min timestamp=1679309841075, max timestamp=1679309841079Num uniques 1069; 2023-03-20 10:57:21,082 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=48954, new segment data size=10024, suffix heap size=139144,new segment heap size=23016 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1069, new segment cells count=219 2023-03-20 10:57:21,082 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=48954, new segment data size=10024, suffix heap size=139144,new segment heap size=23016 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1069, new segment cells count=219 2023-03-20 10:57:21,102 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=228, cellSize=10438, totalHeapSize=23952, min timestamp=1679309841095, max timestamp=1679309841099Num uniques 1078; 2023-03-20 10:57:21,102 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=49368, new segment data size=10438, suffix heap size=140080,new segment heap size=23952 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1078, new segment cells count=228 2023-03-20 10:57:21,102 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=228, cellSize=10438, totalHeapSize=23952, min timestamp=1679309841095, max timestamp=1679309841099Num uniques 1078; 2023-03-20 10:57:21,102 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=228, cellSize=10438, totalHeapSize=23952, min timestamp=1679309841095, max timestamp=1679309841099Num uniques 1078; 2023-03-20 10:57:21,102 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=49368, new segment data size=10438, suffix heap size=140080,new segment heap size=23952 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1078, new segment cells count=228 2023-03-20 10:57:21,102 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=49368, new segment data size=10438, suffix heap size=140080,new segment heap size=23952 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1078, new segment cells count=228 2023-03-20 10:57:21,128 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=237, cellSize=10852, totalHeapSize=24888, min timestamp=1679309841120, max timestamp=1679309841125Num uniques 1087; 2023-03-20 10:57:21,128 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=49782, new segment data size=10852, suffix heap size=141016,new segment heap size=24888 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1087, new segment cells count=237 2023-03-20 10:57:21,129 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=187, cellSize=8562, totalHeapSize=19688, min timestamp=1679309841120, max timestamp=1679309841125Num uniques 1087; 2023-03-20 10:57:21,129 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=49782, new segment data size=8562, suffix heap size=141016,new segment heap size=19688 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1087, new segment cells count=187 2023-03-20 10:57:21,131 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679309841120, max timestamp=1679309841125Num uniques 1087; 2023-03-20 10:57:21,131 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=49782, new segment data size=6870, suffix heap size=141016,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1087, new segment cells count=150 2023-03-20 10:57:21,152 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=246, cellSize=11266, totalHeapSize=25824, min timestamp=1679309841120, max timestamp=1679309841146Num uniques 1096; 2023-03-20 10:57:21,152 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=50196, new segment data size=11266, suffix heap size=141952,new segment heap size=25824 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1096, new segment cells count=246 2023-03-20 10:57:21,153 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=246, cellSize=11266, totalHeapSize=25824, min timestamp=1679309841120, max timestamp=1679309841146Num uniques 1046; 2023-03-20 10:57:21,153 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=47906, new segment data size=11266, suffix heap size=136752,new segment heap size=25824 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1046, new segment cells count=246 2023-03-20 10:57:21,155 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679309841120, max timestamp=1679309841146Num uniques 1009; 2023-03-20 10:57:21,155 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-03-20 10:57:21,190 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=155, cellSize=7095, totalHeapSize=16360, min timestamp=1679309841157, max timestamp=1679309841186Num uniques 1105; 2023-03-20 10:57:21,190 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=155, cellSize=7095, totalHeapSize=16360, min timestamp=1679309841157, max timestamp=1679309841186Num uniques 1105; 2023-03-20 10:57:21,190 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=50605, new segment data size=7095, suffix heap size=142888,new segment heap size=16360 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1105, new segment cells count=155 2023-03-20 10:57:21,190 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=50605, new segment data size=7095, suffix heap size=142888,new segment heap size=16360 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1105, new segment cells count=155 2023-03-20 10:57:21,190 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=155, cellSize=7095, totalHeapSize=16360, min timestamp=1679309841157, max timestamp=1679309841186Num uniques 1009; 2023-03-20 10:57:21,190 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=46209, new segment data size=7095, suffix heap size=132904,new segment heap size=16360 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=155 2023-03-20 10:57:21,208 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=164, cellSize=7504, totalHeapSize=17296, min timestamp=1679309841199, max timestamp=1679309841205Num uniques 1014; 2023-03-20 10:57:21,208 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=46434, new segment data size=7504, suffix heap size=133424,new segment heap size=17296 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1014, new segment cells count=164 2023-03-20 10:57:21,210 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679309841199, max timestamp=1679309841205Num uniques 1014; 2023-03-20 10:57:21,210 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=46434, new segment data size=6870, suffix heap size=133424,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1014, new segment cells count=150 2023-03-20 10:57:21,208 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=214, cellSize=9790, totalHeapSize=22496, min timestamp=1679309841199, max timestamp=1679309841205Num uniques 1014; 2023-03-20 10:57:21,210 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=46430, new segment data size=9790, suffix heap size=133424,new segment heap size=22496 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1014, new segment cells count=214 2023-03-20 10:57:21,214 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=7 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/C/0de5c1a4660d403aaa2ca6c445f40991 2023-03-20 10:57:21,234 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/A/e0f2074420494f18892dd06834b239df as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/e0f2074420494f18892dd06834b239df 2023-03-20 10:57:21,236 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=373, cellSize=17068, totalHeapSize=39032, min timestamp=1679309841217, max timestamp=1679309841233Num uniques 1073; 2023-03-20 10:57:21,236 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=373, cellSize=17078, totalHeapSize=39032, min timestamp=1679309841217, max timestamp=1679309841228Num uniques 1023; 2023-03-20 10:57:21,236 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=46848, new segment data size=17078, suffix heap size=134360,new segment heap size=39032 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1023, new segment cells count=373 2023-03-20 10:57:21,236 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=49128, new segment data size=17068, suffix heap size=139560,new segment heap size=39032 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1073, new segment cells count=373 2023-03-20 10:57:21,237 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=373, cellSize=17078, totalHeapSize=39032, min timestamp=1679309841217, max timestamp=1679309841228Num uniques 1009; 2023-03-20 10:57:21,237 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=17078, suffix heap size=132904,new segment heap size=39032 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=373 2023-03-20 10:57:21,248 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/e0f2074420494f18892dd06834b239df, entries=50, sequenceid=7, filesize=7.0 K 2023-03-20 10:57:21,249 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/B/fc0188b7d1fa415fb217910da8214522 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/fc0188b7d1fa415fb217910da8214522 2023-03-20 10:57:21,260 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/fc0188b7d1fa415fb217910da8214522, entries=50, sequenceid=7, filesize=7.0 K 2023-03-20 10:57:21,261 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/C/0de5c1a4660d403aaa2ca6c445f40991 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/0de5c1a4660d403aaa2ca6c445f40991 2023-03-20 10:57:21,262 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:21,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55136 deadline: 1679309901256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,263 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:21,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55132 deadline: 1679309901258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,262 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:21,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55118 deadline: 1679309901254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,264 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=232, cellSize=10622, totalHeapSize=24368, min timestamp=1679309841249, max timestamp=1679309841260Num uniques 1232; 2023-03-20 10:57:21,264 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=232, cellSize=10622, totalHeapSize=24368, min timestamp=1679309841249, max timestamp=1679309841260Num uniques 1232; 2023-03-20 10:57:21,264 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=56422, new segment data size=10622, suffix heap size=156096,new segment heap size=24368 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1232, new segment cells count=232 2023-03-20 10:57:21,264 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=56422, new segment data size=10622, suffix heap size=156096,new segment heap size=24368 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1232, new segment cells count=232 2023-03-20 10:57:21,265 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679309841253, max timestamp=1679309841260Num uniques 1232; 2023-03-20 10:57:21,265 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=56422, new segment data size=6870, suffix heap size=156096,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1232, new segment cells count=150 2023-03-20 10:57:21,277 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/0de5c1a4660d403aaa2ca6c445f40991, entries=50, sequenceid=7, filesize=7.0 K 2023-03-20 10:57:21,292 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=50.01 KB/51208 for 72402f7d20e803d356627e208deb6fe2 in 1382ms, sequenceid=7, compaction requested=false 2023-03-20 10:57:21,292 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-03-20 10:57:21,293 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 72402f7d20e803d356627e208deb6fe2: 2023-03-20 10:57:21,293 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:21,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 72402f7d20e803d356627e208deb6fe2 2023-03-20 10:57:21,295 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 72402f7d20e803d356627e208deb6fe2 3/3 column families, dataSize=63.43 KB heapSize=170.59 KB 2023-03-20 10:57:21,296 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=A 2023-03-20 10:57:21,296 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:57:21,296 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=B 2023-03-20 10:57:21,297 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:57:21,297 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=C 2023-03-20 10:57:21,297 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:57:21,298 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:21,298 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:21,298 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,298 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:21,298 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:21,299 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:21,299 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:21,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:21,300 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:21,300 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:21,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:21,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:21,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:21,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:21,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:21,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24f72c04[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:21,302 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:21,302 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:21,303 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:21,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:21,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:21,304 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:21,304 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:21,304 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,304 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:21,304 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:21,305 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:21,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:21,305 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:21,305 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:21,305 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:21,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:21,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:21,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:21,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:21,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:21,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:21,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:21,308 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:21,308 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:21,308 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:21,308 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:21,308 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:21,310 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:21,310 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:21,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:21,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:21,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:21,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:21,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:21,313 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:21,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:21,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:21,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:21,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:21,314 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:21,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:21,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:21,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,316 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:21,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:21,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,318 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:21,318 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:21,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:21,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:21,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:21,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:21,319 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,319 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:21,319 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:21,319 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,319 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:21,319 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:21,319 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:21,320 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:21,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:21,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:21,320 DEBUG [Thread-1564] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:21,320 DEBUG [Thread-1564] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:21,356 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1679309841332, max timestamp=1679309841353Num uniques 859; 2023-03-20 10:57:21,356 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-20 10:57:21,358 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679309841332, max timestamp=1679309841353Num uniques 859; 2023-03-20 10:57:21,358 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-20 10:57:21,358 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679309841332, max timestamp=1679309841353Num uniques 859; 2023-03-20 10:57:21,358 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-20 10:57:21,384 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:21,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55182 deadline: 1679309901379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,385 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:21,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55132 deadline: 1679309901380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,385 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:21,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55168 deadline: 1679309901381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,385 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:21,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55118 deadline: 1679309901382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,387 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:21,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55136 deadline: 1679309901384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,420 DEBUG [Thread-1564] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:21,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:21,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:21,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:21,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:21,427 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:21,427 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:21,427 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:21,427 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:21,427 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:21,427 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:21,428 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,428 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,428 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:21,430 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:21,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:21,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:21,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:21,431 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:21,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:21,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:21,431 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:21,431 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:21,431 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:21,431 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:21,431 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:21,431 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:21,432 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:21,432 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:21,432 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:21,432 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. started... 2023-03-20 10:57:21,432 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. as already flushing 2023-03-20 10:57:21,432 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:21,432 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:21,432 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:21,432 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,432 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:21,432 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:21,434 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:21,434 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,434 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:21,434 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:21,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:21,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:21,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:21,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:21,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:21,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:21,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:21,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c1af2f8[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:21,436 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:21,436 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:21,437 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:21,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:21,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:21,437 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:21,437 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:21,437 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:21,437 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,437 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:21,439 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,439 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:21,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:21,439 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:21,440 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:21,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:21,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:21,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:21,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:21,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:21,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:21,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:21,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:21,443 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:21,443 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:21,443 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:21,443 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:21,443 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:21,453 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:21,453 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:21,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:21,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:21,458 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:21,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:21,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:21,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:21,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:21,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:21,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:21,459 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:21,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:21,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:21,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:21,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:21,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:21,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,469 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,472 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:21,472 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:21,472 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:21,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:21,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:21,473 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:21,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:21,473 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,473 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:21,473 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:21,473 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,473 DEBUG [Thread-1564] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:21,473 DEBUG [Thread-1564] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:21,473 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:21,473 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:21,473 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:21,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:21,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:21,488 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:21,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55182 deadline: 1679309901486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,488 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:21,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55132 deadline: 1679309901486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,489 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:21,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55168 deadline: 1679309901487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,489 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:21,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55118 deadline: 1679309901487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,490 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:21,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55136 deadline: 1679309901488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,573 DEBUG [Thread-1564] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:21,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:21,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:21,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:21,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:21,578 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:21,578 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:21,578 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:21,578 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:21,578 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:21,578 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:21,579 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:21,579 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:21,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:21,580 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,580 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:21,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:21,580 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:21,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:21,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:21,581 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:21,582 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:21,582 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:21,582 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:21,582 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:21,582 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:21,582 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:21,582 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:21,582 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:21,582 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. started... 2023-03-20 10:57:21,583 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. as already flushing 2023-03-20 10:57:21,583 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:21,583 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:21,583 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:21,583 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,583 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:21,583 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:21,584 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:21,584 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:21,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:21,584 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:21,584 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:21,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:21,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:21,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:21,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:21,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:21,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12708a59[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:21,586 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:21,586 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:21,588 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:21,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:21,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:21,588 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,588 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:21,589 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:21,588 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:21,589 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:21,589 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,589 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:21,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:21,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:21,589 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:21,590 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:21,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:21,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:21,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:21,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:21,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:21,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:21,593 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:21,593 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:21,593 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:21,593 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:21,593 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:21,593 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:21,594 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:21,594 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:21,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:21,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:21,594 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:21,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:21,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:21,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:21,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:21,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:21,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:21,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:21,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:21,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:21,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:21,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:21,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:21,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,598 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,599 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,600 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:21,600 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:21,600 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,600 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:21,600 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:21,601 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:21,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:21,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:21,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:21,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:21,601 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,601 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:21,601 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:21,601 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:21,601 DEBUG [Thread-1564] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:21,601 DEBUG [Thread-1564] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:21,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:21,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:21,690 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:21,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55132 deadline: 1679309901689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,691 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:21,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55168 deadline: 1679309901690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,692 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:21,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55118 deadline: 1679309901690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,692 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:21,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55182 deadline: 1679309901690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,694 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:21,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55136 deadline: 1679309901692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,701 DEBUG [Thread-1564] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:21,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:21,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:21,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:21,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:21,710 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:21,710 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:21,710 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:21,710 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:21,711 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:21,711 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:21,711 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:21,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:21,711 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:21,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:21,712 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,712 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:21,712 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:21,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:21,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:21,712 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:21,714 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:21,714 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=840 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/A/edc884406ce34f9f9441ff0cd73956ba 2023-03-20 10:57:21,714 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:21,715 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:21,715 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:21,715 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:21,722 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:21,722 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:21,722 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:21,722 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. started... 2023-03-20 10:57:21,724 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. as already flushing 2023-03-20 10:57:21,724 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:21,724 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:21,724 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:21,724 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,724 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:21,724 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:21,728 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:21,728 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:21,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:21,729 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:21,729 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:21,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:21,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:21,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:21,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:21,730 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:21,730 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35165e66[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:21,730 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:21,730 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:21,731 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:21,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:21,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:21,731 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:21,731 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:21,731 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:21,731 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,731 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:21,732 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,732 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:21,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:21,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:21,732 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:21,732 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:21,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:21,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:21,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:21,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:21,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:21,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:21,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:21,734 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:21,734 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:21,734 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:21,734 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:21,734 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:21,736 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=840 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/B/267a75d458d944a9a9304cef4a9881cb 2023-03-20 10:57:21,736 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:21,736 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:21,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:21,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:21,736 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:21,737 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:21,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:21,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:21,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:21,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:21,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:21,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:21,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:21,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:21,741 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:21,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:21,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:21,745 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,745 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:21,745 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:21,745 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,745 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:21,745 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:21,745 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:21,745 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:21,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:21,746 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:57:21,746 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:21,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:21,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:21,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:21,746 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:21,746 DEBUG [Thread-1564] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:21,746 DEBUG [Thread-1564] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:21,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:21,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:21,846 DEBUG [Thread-1564] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:21,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:21,848 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:21,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:21,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:21,850 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:21,850 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:21,850 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:21,850 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:21,851 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:21,851 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:21,851 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:21,851 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:21,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:21,852 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,852 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:21,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:21,852 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:21,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:21,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:21,853 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:21,853 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:21,853 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:21,853 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:21,853 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:21,853 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:21,854 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:21,854 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:21,854 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:21,854 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. started... 2023-03-20 10:57:21,854 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. as already flushing 2023-03-20 10:57:21,854 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:21,854 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:21,854 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:21,854 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,854 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:21,854 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:21,855 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:21,855 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:21,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:21,856 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:21,856 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:21,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:21,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:21,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:21,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:21,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:21,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@281cd3cc[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:21,857 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:21,858 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:21,859 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:21,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:21,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:21,860 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:21,860 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:21,860 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:21,860 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,860 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:21,863 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:21,863 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:21,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:21,863 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:21,863 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:21,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:21,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:21,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:21,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:21,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:21,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:21,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:21,865 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:21,866 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:21,866 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:21,866 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:21,866 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:21,870 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:21,870 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:21,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:21,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:21,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:21,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:21,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:21,870 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:21,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:21,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:21,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:21,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:21,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:21,872 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:21,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:21,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:21,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:21,873 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,879 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,879 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:21,879 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:21,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:21,879 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:21,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:21,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:21,879 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:21,879 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:21,879 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,879 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:21,879 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:21,879 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:21,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:21,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:21,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:21,880 DEBUG [Thread-1564] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:21,880 DEBUG [Thread-1564] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:21,980 DEBUG [Thread-1564] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:21,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:21,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:21,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:21,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:21,984 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:21,984 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:21,984 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:21,984 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:21,984 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:21,984 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:21,985 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,985 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:21,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:21,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:21,985 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,985 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:21,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:21,985 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:21,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:21,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:21,986 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:21,986 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:21,986 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:21,986 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:21,986 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:21,986 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:21,990 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:21,990 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:21,990 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:21,990 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. started... 2023-03-20 10:57:21,990 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. as already flushing 2023-03-20 10:57:21,990 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:21,990 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:21,990 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:21,990 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,990 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:21,990 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:21,994 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:21,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55168 deadline: 1679309901992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,995 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:21,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55132 deadline: 1679309901992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,995 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:21,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55182 deadline: 1679309901993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,995 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:21,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55118 deadline: 1679309901993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:21,998 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:21,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55136 deadline: 1679309901997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:22,003 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:22,004 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:22,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:22,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:22,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:22,004 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:22,004 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:22,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:22,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:22,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:22,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:22,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:22,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:22,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43fa6905[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:22,005 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:22,005 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:22,011 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:22,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:22,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:22,011 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:22,012 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:22,012 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:22,012 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:22,012 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:22,014 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:22,014 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:22,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:22,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:22,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:22,014 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:22,014 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:22,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:22,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:22,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:22,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:22,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:22,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:22,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:22,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:22,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:22,016 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:22,016 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:22,016 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:22,016 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:22,016 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:22,018 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:22,018 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:22,018 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:22,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:22,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:22,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:22,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:22,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:22,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:22,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:22,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:22,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:22,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:22,021 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:22,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:22,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:22,023 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:22,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:22,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:22,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:22,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:22,026 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:22,026 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:22,026 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:22,026 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:22,026 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:22,026 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:22,026 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:22,026 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:22,026 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:22,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:22,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:22,026 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:57:22,026 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:22,026 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:22,026 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:22,026 DEBUG [Thread-1564] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:22,026 DEBUG [Thread-1564] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:22,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:22,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:22,127 DEBUG [Thread-1564] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:22,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:22,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:22,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:22,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:22,148 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:22,148 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:22,148 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:22,148 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:22,149 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:22,149 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:22,150 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:22,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:22,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:22,150 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:22,150 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:22,150 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:22,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:22,151 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:22,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:22,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:22,151 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:22,151 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=18.70 KB at sequenceid=840 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/C/f88cb61c68f646cbbf1eea0284a24779 2023-03-20 10:57:22,156 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/A/edc884406ce34f9f9441ff0cd73956ba as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/edc884406ce34f9f9441ff0cd73956ba 2023-03-20 10:57:22,162 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/edc884406ce34f9f9441ff0cd73956ba, entries=150, sequenceid=840, filesize=11.9 K 2023-03-20 10:57:22,162 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:22,162 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:22,162 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/B/267a75d458d944a9a9304cef4a9881cb as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/267a75d458d944a9a9304cef4a9881cb 2023-03-20 10:57:22,162 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:22,163 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:22,163 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:22,166 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/267a75d458d944a9a9304cef4a9881cb, entries=150, sequenceid=840, filesize=11.9 K 2023-03-20 10:57:22,172 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/C/f88cb61c68f646cbbf1eea0284a24779 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/f88cb61c68f646cbbf1eea0284a24779 2023-03-20 10:57:22,175 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:22,175 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:22,175 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:22,175 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. started... 2023-03-20 10:57:22,175 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. as already flushing 2023-03-20 10:57:22,175 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:22,176 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:22,176 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:22,176 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:22,176 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:22,176 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:22,178 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:22,178 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:22,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:22,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:22,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:22,178 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:22,178 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:22,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:22,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:22,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:22,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:22,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:22,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:22,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55ce0161[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:22,180 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:22,180 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:22,181 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:22,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:22,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:22,181 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:22,181 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:22,181 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:22,181 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:22,181 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:22,182 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:22,182 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:22,182 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:22,183 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:22,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:22,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:22,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:22,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:22,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:22,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:22,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:22,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:22,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:22,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:22,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:22,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:22,185 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:22,185 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:22,185 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:22,185 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:22,185 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:22,187 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:22,187 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:22,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:22,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:22,187 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:22,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:22,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:22,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:22,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:22,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:22,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:22,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:22,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:22,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:22,190 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:22,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:22,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:22,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:22,191 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:22,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:22,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:22,195 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/f88cb61c68f646cbbf1eea0284a24779, entries=150, sequenceid=840, filesize=11.9 K 2023-03-20 10:57:22,195 DEBUG [MemStoreFlusher.0] regionserver.StoreScanner(953): StoreScanner already has the close lock. There is no need to updateReaders 2023-03-20 10:57:22,195 DEBUG [MemStoreFlusher.0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-03-20 10:57:22,201 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~63.43 KB/64948, heapSize ~170.55 KB/174640, currentSize=123.00 KB/125950 for 72402f7d20e803d356627e208deb6fe2 in 906ms, sequenceid=840, compaction requested=false 2023-03-20 10:57:22,203 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 72402f7d20e803d356627e208deb6fe2: 2023-03-20 10:57:22,204 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:22,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:22,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:22,204 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:22,204 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:22,204 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:22,204 DEBUG [Thread-1564] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:22,205 DEBUG [Thread-1564] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:22,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:22,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:22,204 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:22,205 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:22,205 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:22,205 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:22,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:22,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:22,205 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:22,206 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:22,305 DEBUG [Thread-1564] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:22,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:22,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:22,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:22,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:22,313 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:22,314 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:22,314 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:22,314 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:22,314 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:22,314 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:22,316 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:22,316 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:22,316 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:22,316 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:22,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:22,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:22,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:22,317 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:22,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:22,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:22,317 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:22,318 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:22,318 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:22,318 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:22,318 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:22,318 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:22,322 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:22,322 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:22,323 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:22,323 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. started... 2023-03-20 10:57:22,323 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 72402f7d20e803d356627e208deb6fe2 3/3 column families, dataSize=123.00 KB heapSize=350.22 KB 2023-03-20 10:57:22,323 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=A 2023-03-20 10:57:22,323 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:57:22,323 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=B 2023-03-20 10:57:22,323 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:57:22,323 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=C 2023-03-20 10:57:22,324 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:57:22,348 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=875 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/A/0d344dfab12d46749d37c0a4e6b84600 2023-03-20 10:57:22,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 72402f7d20e803d356627e208deb6fe2 2023-03-20 10:57:22,498 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. as already flushing 2023-03-20 10:57:22,507 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:22,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55168 deadline: 1679309902506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:22,509 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:22,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55136 deadline: 1679309902507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:22,510 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:22,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55182 deadline: 1679309902507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:22,511 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:22,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55132 deadline: 1679309902507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:22,514 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:22,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55118 deadline: 1679309902512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:22,610 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:22,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55168 deadline: 1679309902608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:22,612 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:22,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55136 deadline: 1679309902610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:22,613 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:22,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55132 deadline: 1679309902611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:22,613 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:22,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55182 deadline: 1679309902611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:22,780 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=875 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/B/bc6fd36293f34956a497b5a66c4f7507 2023-03-20 10:57:22,813 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:22,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55168 deadline: 1679309902812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:22,816 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:22,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55136 deadline: 1679309902814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:22,816 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:22,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55132 deadline: 1679309902814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:22,816 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:22,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55182 deadline: 1679309902814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:23,117 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:23,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55168 deadline: 1679309903114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:23,119 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:23,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55136 deadline: 1679309903117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:23,121 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:23,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55182 deadline: 1679309903118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:23,131 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:23,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55132 deadline: 1679309903129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:23,196 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=875 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/C/40b1a0810c8e4ee3a04c8cdb6c180d49 2023-03-20 10:57:23,205 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/A/0d344dfab12d46749d37c0a4e6b84600 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/0d344dfab12d46749d37c0a4e6b84600 2023-03-20 10:57:23,216 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/0d344dfab12d46749d37c0a4e6b84600, entries=150, sequenceid=875, filesize=11.9 K 2023-03-20 10:57:23,218 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/B/bc6fd36293f34956a497b5a66c4f7507 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/bc6fd36293f34956a497b5a66c4f7507 2023-03-20 10:57:23,228 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/bc6fd36293f34956a497b5a66c4f7507, entries=150, sequenceid=875, filesize=11.9 K 2023-03-20 10:57:23,231 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/C/40b1a0810c8e4ee3a04c8cdb6c180d49 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/40b1a0810c8e4ee3a04c8cdb6c180d49 2023-03-20 10:57:23,246 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/40b1a0810c8e4ee3a04c8cdb6c180d49, entries=150, sequenceid=875, filesize=11.9 K 2023-03-20 10:57:23,247 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~123.00 KB/125950, heapSize ~350.17 KB/358576, currentSize=67.09 KB/68700 for 72402f7d20e803d356627e208deb6fe2 in 924ms, sequenceid=875, compaction requested=true 2023-03-20 10:57:23,248 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 72402f7d20e803d356627e208deb6fe2: 2023-03-20 10:57:23,248 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:23,248 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:23,248 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:23,248 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:23,248 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:23,248 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:23,249 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:23,249 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:23,249 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:23,250 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:23,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:23,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:23,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:23,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:23,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:23,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:23,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:23,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:23,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:23,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@522c80d0[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:23,253 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:23,253 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:23,254 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:23,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:23,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:23,254 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:23,254 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:23,254 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:23,254 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:23,254 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:23,255 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:23,255 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:23,255 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:23,258 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:23,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:23,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:23,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:23,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:23,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:23,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:23,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:23,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:23,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:23,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:23,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:23,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:23,262 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:23,262 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:23,262 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:23,262 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:23,262 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:23,271 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:23,271 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:23,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:23,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:23,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:23,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:23,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:23,271 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:23,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:23,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:23,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:23,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:23,273 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:23,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:23,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:23,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:23,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:23,274 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:23,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:23,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:23,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:23,279 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:23,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:23,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:23,279 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:23,279 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:23,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:23,279 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:23,279 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:23,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:23,279 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:23,279 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:23,279 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:23,279 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:23,279 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:23,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:23,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:23,282 DEBUG [Thread-1564] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:23,282 DEBUG [Thread-1564] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:23,382 DEBUG [Thread-1564] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:23,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:23,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:23,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:23,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:23,385 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:23,385 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:23,385 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:23,385 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:23,385 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:23,385 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:23,449 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-20 10:57:23,481 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:23,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:23,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:23,481 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:23,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:23,482 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:23,482 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:23,482 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:23,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:23,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:23,483 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:23,485 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:23,485 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:23,485 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:23,485 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:23,485 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:23,485 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:23,485 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:23,485 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:23,485 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. started... 2023-03-20 10:57:23,486 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 72402f7d20e803d356627e208deb6fe2 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-20 10:57:23,486 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=A 2023-03-20 10:57:23,486 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:23,486 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=B 2023-03-20 10:57:23,487 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:23,487 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=C 2023-03-20 10:57:23,487 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:23,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 72402f7d20e803d356627e208deb6fe2 2023-03-20 10:57:23,528 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. as already flushing 2023-03-20 10:57:23,588 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:23,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55118 deadline: 1679309903585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:23,645 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:23,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55132 deadline: 1679309903641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:23,645 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:23,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55168 deadline: 1679309903642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:23,645 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:23,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55136 deadline: 1679309903642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:23,645 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:23,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55182 deadline: 1679309903642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:23,693 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:23,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55118 deadline: 1679309903692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:23,898 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:23,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55118 deadline: 1679309903896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:23,919 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=888 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/A/4f8b9b806fb64895b4075ab6dfdf4928 2023-03-20 10:57:24,202 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:24,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55118 deadline: 1679309904201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:24,340 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=888 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/B/5513e593024f49da943e0691c383825c 2023-03-20 10:57:24,351 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=888 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/C/40e5c692a5cd4e958aec77a681dbe2da 2023-03-20 10:57:24,355 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/A/4f8b9b806fb64895b4075ab6dfdf4928 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/4f8b9b806fb64895b4075ab6dfdf4928 2023-03-20 10:57:24,361 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/4f8b9b806fb64895b4075ab6dfdf4928, entries=150, sequenceid=888, filesize=11.9 K 2023-03-20 10:57:24,361 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/B/5513e593024f49da943e0691c383825c as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/5513e593024f49da943e0691c383825c 2023-03-20 10:57:24,365 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/5513e593024f49da943e0691c383825c, entries=150, sequenceid=888, filesize=11.9 K 2023-03-20 10:57:24,366 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/C/40e5c692a5cd4e958aec77a681dbe2da as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/40e5c692a5cd4e958aec77a681dbe2da 2023-03-20 10:57:24,369 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/40e5c692a5cd4e958aec77a681dbe2da, entries=150, sequenceid=888, filesize=11.9 K 2023-03-20 10:57:24,370 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 72402f7d20e803d356627e208deb6fe2 in 885ms, sequenceid=888, compaction requested=true 2023-03-20 10:57:24,370 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 72402f7d20e803d356627e208deb6fe2: 2023-03-20 10:57:24,370 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:24,370 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:24,370 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:24,370 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:24,370 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:24,370 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:24,371 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:24,371 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:24,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:24,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:24,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:24,371 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:24,371 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:24,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:24,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:24,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:24,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:24,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:24,372 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:24,372 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c60bbea[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:24,372 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:24,373 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:24,373 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:24,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:24,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:24,373 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:24,373 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:24,373 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:24,373 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:24,373 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:24,374 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:24,374 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:24,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:24,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:24,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:24,374 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:24,374 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:24,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:24,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:24,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:24,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:24,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:24,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:24,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:24,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:24,377 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:24,377 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:24,377 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:24,377 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:24,377 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:24,377 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:24,378 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:24,378 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:24,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:24,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:24,378 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:24,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:24,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:24,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:24,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:24,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:24,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:24,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:24,379 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:24,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:24,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:24,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:24,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:24,380 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:24,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:24,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:24,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:24,382 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:24,382 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:24,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:24,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:24,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:24,382 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:24,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:24,383 DEBUG [Thread-1564] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:24,383 DEBUG [Thread-1564] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:24,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:24,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:24,385 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:24,385 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:24,385 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:24,385 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:24,385 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:24,385 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:24,385 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:24,483 DEBUG [Thread-1564] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:24,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:24,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:24,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:24,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:24,485 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:24,485 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:24,485 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:24,485 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:24,486 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:24,486 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:24,486 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:24,486 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:24,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:24,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:24,486 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:24,486 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:24,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:24,487 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:24,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:24,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:24,487 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:24,487 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:24,487 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:24,487 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:24,487 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:24,487 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:24,487 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:24,488 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:24,488 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:24,488 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. started... 2023-03-20 10:57:24,488 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 72402f7d20e803d356627e208deb6fe2 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-20 10:57:24,488 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=A 2023-03-20 10:57:24,488 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:24,488 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=B 2023-03-20 10:57:24,488 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:24,488 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=C 2023-03-20 10:57:24,488 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:24,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 72402f7d20e803d356627e208deb6fe2 2023-03-20 10:57:24,652 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. as already flushing 2023-03-20 10:57:24,664 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:24,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55182 deadline: 1679309904659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:24,664 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:24,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55132 deadline: 1679309904660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:24,665 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:24,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55168 deadline: 1679309904663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:24,669 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:24,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55136 deadline: 1679309904668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:24,710 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:24,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55118 deadline: 1679309904707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:24,766 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:24,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55132 deadline: 1679309904765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:24,770 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:24,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55168 deadline: 1679309904767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:24,771 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:24,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55136 deadline: 1679309904770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:24,774 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:24,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55182 deadline: 1679309904771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:24,900 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=907 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/A/807d4f3ae4514bb8b8b91ff8e52a7776 2023-03-20 10:57:24,913 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=907 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/B/e967d0bedf4a47789400ecc5dd7c2ad2 2023-03-20 10:57:24,971 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:24,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55132 deadline: 1679309904969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:24,973 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:24,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55168 deadline: 1679309904971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:24,974 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:24,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55136 deadline: 1679309904973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:24,977 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:24,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55182 deadline: 1679309904976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:25,275 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:25,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55132 deadline: 1679309905273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:25,277 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:25,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55168 deadline: 1679309905274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:25,279 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:25,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55136 deadline: 1679309905276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:25,281 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:25,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55182 deadline: 1679309905279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:25,322 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=907 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/C/5a1be9933de24bc4acf224f5fd026dbd 2023-03-20 10:57:25,333 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/A/807d4f3ae4514bb8b8b91ff8e52a7776 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/807d4f3ae4514bb8b8b91ff8e52a7776 2023-03-20 10:57:25,349 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/807d4f3ae4514bb8b8b91ff8e52a7776, entries=150, sequenceid=907, filesize=11.9 K 2023-03-20 10:57:25,353 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/B/e967d0bedf4a47789400ecc5dd7c2ad2 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/e967d0bedf4a47789400ecc5dd7c2ad2 2023-03-20 10:57:25,373 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/e967d0bedf4a47789400ecc5dd7c2ad2, entries=150, sequenceid=907, filesize=11.9 K 2023-03-20 10:57:25,374 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/C/5a1be9933de24bc4acf224f5fd026dbd as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/5a1be9933de24bc4acf224f5fd026dbd 2023-03-20 10:57:25,380 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/5a1be9933de24bc4acf224f5fd026dbd, entries=150, sequenceid=907, filesize=11.9 K 2023-03-20 10:57:25,381 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 72402f7d20e803d356627e208deb6fe2 in 893ms, sequenceid=907, compaction requested=true 2023-03-20 10:57:25,381 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 72402f7d20e803d356627e208deb6fe2: 2023-03-20 10:57:25,381 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:25,381 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:25,381 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:25,381 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:25,381 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:25,381 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:25,383 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:25,383 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:25,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:25,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:25,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:25,383 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:25,383 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:25,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:25,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:25,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:25,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:25,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:25,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:25,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c425b22[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:25,386 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:25,386 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:25,387 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:25,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:25,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:25,387 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:25,387 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:25,387 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:25,388 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:25,388 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:25,388 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:25,393 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:25,393 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:25,393 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:25,393 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:25,393 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:25,393 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:25,393 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:25,393 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:25,403 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:25,403 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:25,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:25,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:25,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:25,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:25,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:25,407 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:25,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:25,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:25,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:25,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:25,407 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:25,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:25,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:25,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:25,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:25,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:25,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:25,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:25,415 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:25,417 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:25,417 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:25,417 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:25,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:25,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:25,417 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:25,417 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:25,417 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:25,417 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:25,417 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:25,417 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:25,417 DEBUG [Thread-1564] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:25,417 DEBUG [Thread-1564] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:25,417 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:25,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:25,417 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:25,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:25,418 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:25,517 DEBUG [Thread-1564] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:25,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:25,520 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:25,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:25,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:25,538 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:25,538 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:25,539 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:25,539 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:25,539 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:25,539 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:25,540 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:25,540 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:25,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:25,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:25,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:25,540 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:25,540 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:25,541 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:25,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:25,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:25,541 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:25,542 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:25,542 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:25,542 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:25,542 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:25,542 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:25,553 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:25,553 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:25,566 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:25,566 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. started... 2023-03-20 10:57:25,566 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 72402f7d20e803d356627e208deb6fe2 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-20 10:57:25,566 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=A 2023-03-20 10:57:25,567 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:25,567 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=B 2023-03-20 10:57:25,567 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:25,567 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=C 2023-03-20 10:57:25,567 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:25,638 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=920 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/A/612f5f5d981543779183450ac9c6e51b 2023-03-20 10:57:25,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 72402f7d20e803d356627e208deb6fe2 2023-03-20 10:57:25,720 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. as already flushing 2023-03-20 10:57:25,779 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:25,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55132 deadline: 1679309905776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:25,782 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:25,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55118 deadline: 1679309905779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:25,783 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:25,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55168 deadline: 1679309905779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:25,784 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:25,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55136 deadline: 1679309905781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:25,789 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:25,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55182 deadline: 1679309905786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:25,888 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:25,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55118 deadline: 1679309905886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:26,069 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=920 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/B/ecbb1475a4fe40779070fd1a43948990 2023-03-20 10:57:26,094 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:26,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55118 deadline: 1679309906089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:26,402 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:26,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55118 deadline: 1679309906397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:26,586 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=920 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/C/e1a7d61b163b46ed8c107c37f9b782d8 2023-03-20 10:57:26,592 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/A/612f5f5d981543779183450ac9c6e51b as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/612f5f5d981543779183450ac9c6e51b 2023-03-20 10:57:26,598 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/612f5f5d981543779183450ac9c6e51b, entries=150, sequenceid=920, filesize=11.9 K 2023-03-20 10:57:26,599 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/B/ecbb1475a4fe40779070fd1a43948990 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/ecbb1475a4fe40779070fd1a43948990 2023-03-20 10:57:26,604 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/ecbb1475a4fe40779070fd1a43948990, entries=150, sequenceid=920, filesize=11.9 K 2023-03-20 10:57:26,605 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/C/e1a7d61b163b46ed8c107c37f9b782d8 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/e1a7d61b163b46ed8c107c37f9b782d8 2023-03-20 10:57:26,619 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/e1a7d61b163b46ed8c107c37f9b782d8, entries=150, sequenceid=920, filesize=11.9 K 2023-03-20 10:57:26,621 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 72402f7d20e803d356627e208deb6fe2 in 1055ms, sequenceid=920, compaction requested=true 2023-03-20 10:57:26,621 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 72402f7d20e803d356627e208deb6fe2: 2023-03-20 10:57:26,621 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:26,621 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:26,621 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:26,621 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:26,621 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:26,622 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:26,623 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:26,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:26,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:26,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:26,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:26,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:26,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:26,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:26,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:26,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:26,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29a296c2[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:26,625 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:26,625 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:26,625 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:26,625 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:26,626 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:26,626 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:26,626 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:26,626 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:26,626 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:26,626 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:26,626 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:26,628 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:26,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:26,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:26,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:26,628 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:26,628 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:26,628 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:26,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:26,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:26,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:26,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:26,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:26,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:26,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:26,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:26,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:26,639 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:26,639 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:26,639 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:26,639 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:26,639 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:26,645 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:26,645 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:26,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:26,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:26,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:26,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:26,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:26,645 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:26,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:26,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:26,647 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:26,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:26,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:26,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:26,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:26,649 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:26,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:26,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:26,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:26,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:26,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:26,653 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:26,654 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:26,654 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:26,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:26,654 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:26,654 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:26,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:26,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:26,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:26,654 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:26,654 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:26,654 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:26,654 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:26,654 DEBUG [Thread-1564] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:26,654 DEBUG [Thread-1564] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:26,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:26,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:26,654 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:26,655 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:26,655 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:57:26,754 DEBUG [Thread-1564] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:26,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:26,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:26,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:26,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:26,787 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:26,788 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:26,788 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:26,788 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:26,788 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:26,788 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:26,789 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:26,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:26,789 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:26,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:26,789 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:26,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:26,789 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:26,790 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:26,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:26,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:26,790 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:26,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 72402f7d20e803d356627e208deb6fe2 2023-03-20 10:57:26,791 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 72402f7d20e803d356627e208deb6fe2 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-20 10:57:26,791 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=A 2023-03-20 10:57:26,791 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:57:26,791 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=B 2023-03-20 10:57:26,791 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:57:26,791 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=C 2023-03-20 10:57:26,792 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:57:26,796 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:26,796 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:26,796 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:26,796 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:26,797 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:26,800 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:26,800 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:26,800 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:26,800 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. started... 2023-03-20 10:57:26,801 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. as already flushing 2023-03-20 10:57:26,801 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:26,801 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:26,801 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:26,801 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:26,806 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:26,806 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:26,807 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:26,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55132 deadline: 1679309906802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:26,807 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:26,807 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:26,807 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:26,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:26,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55168 deadline: 1679309906804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:26,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:26,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:26,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:26,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:26,808 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:26,808 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:26,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:26,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:26,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:26,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:26,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1fbeb926[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:26,811 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:26,811 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:26,811 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:26,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:26,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:26,811 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:26,811 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:26,812 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:26,811 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:26,812 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:26,812 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:26,812 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:26,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:26,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:26,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:26,812 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:26,812 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:26,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:26,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:26,813 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:26,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:26,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55182 deadline: 1679309906808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:26,813 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:26,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55136 deadline: 1679309906808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:26,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:26,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:26,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:26,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:26,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:26,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:26,819 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:26,819 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:26,819 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:26,819 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:26,819 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:26,819 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=941 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/A/6c802ff1c7604d9fb67fa2d19a4e97e5 2023-03-20 10:57:26,820 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:26,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:26,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:26,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:26,821 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:26,821 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:26,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:26,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:26,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:26,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:26,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:26,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:26,823 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:26,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:26,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:26,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:26,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:26,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:26,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:26,833 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:26,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:26,846 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:26,846 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:26,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:26,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:26,847 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:26,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:26,847 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:26,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:26,847 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:26,847 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:26,847 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:26,847 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:26,847 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:26,847 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:26,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:26,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:26,847 DEBUG [Thread-1564] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:26,847 DEBUG [Thread-1564] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:26,911 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:26,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55118 deadline: 1679309906908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:26,913 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:26,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55132 deadline: 1679309906909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:26,914 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:26,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55168 deadline: 1679309906910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:26,919 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:26,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55136 deadline: 1679309906916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:26,920 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:26,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55182 deadline: 1679309906917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:26,947 DEBUG [Thread-1564] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:26,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:26,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:26,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:26,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:26,962 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:26,962 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:26,962 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:26,962 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:26,963 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:26,963 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:26,970 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:26,970 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:26,971 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:26,971 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:26,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:26,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:26,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:26,986 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:26,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:26,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:26,987 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:26,998 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:26,998 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:26,998 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:26,998 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:26,998 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:27,014 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:27,014 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:27,014 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:27,014 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. started... 2023-03-20 10:57:27,014 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. as already flushing 2023-03-20 10:57:27,014 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:27,014 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:27,014 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:27,015 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,015 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:27,015 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:27,026 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:27,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:27,027 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:27,027 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:27,027 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:27,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:27,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:27,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:27,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:27,030 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:27,030 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@698eed0[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:27,030 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:27,030 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:27,036 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:27,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:27,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:27,036 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:27,036 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:27,036 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:27,037 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,037 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:27,038 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:27,038 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,038 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:27,038 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:27,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:27,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:27,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:27,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:27,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:27,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:27,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:27,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:27,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:27,041 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:27,041 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:27,041 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:27,041 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:27,041 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:27,042 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:27,042 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:27,042 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:27,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:27,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:27,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:27,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:27,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:27,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:27,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:27,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:27,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:27,044 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:27,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:27,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:27,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,045 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:27,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:27,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,053 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:27,054 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:27,054 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:27,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:27,054 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:57:27,054 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:27,054 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:27,054 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,054 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:27,054 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:27,054 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:27,054 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:27,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:27,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:27,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:27,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:27,055 DEBUG [Thread-1564] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:27,055 DEBUG [Thread-1564] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:27,115 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:27,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55132 deadline: 1679309907115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,121 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:27,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55168 deadline: 1679309907116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,126 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:27,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55136 deadline: 1679309907122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,126 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:27,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55182 deadline: 1679309907122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,155 DEBUG [Thread-1564] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:27,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:27,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:27,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:27,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:27,164 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:27,164 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:27,164 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:27,164 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:27,164 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:27,164 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:27,165 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:27,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:27,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:27,165 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:27,166 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,166 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:27,166 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:27,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:27,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:27,167 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:27,167 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:27,167 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:27,167 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:27,167 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:27,167 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:27,167 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:27,167 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:27,167 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:27,167 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. started... 2023-03-20 10:57:27,167 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. as already flushing 2023-03-20 10:57:27,167 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:27,167 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:27,167 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:27,168 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,168 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:27,168 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:27,169 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:27,169 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:27,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:27,169 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:27,169 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:27,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:27,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:27,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:27,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:27,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:27,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@181d2e6a[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:27,171 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:27,171 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:27,172 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:27,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:27,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:27,172 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:27,173 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:27,173 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:27,173 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,173 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:27,173 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,173 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:27,173 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:27,173 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:27,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:27,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:27,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:27,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:27,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:27,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:27,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:27,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:27,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:27,176 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:27,176 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:27,176 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:27,176 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:27,176 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:27,177 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:27,177 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:27,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:27,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:27,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:27,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:27,178 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:27,178 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:27,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:27,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:27,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:27,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:27,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:27,180 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:27,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:27,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:27,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:27,182 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,182 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:27,182 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:27,182 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,182 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:27,182 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:27,182 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:27,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:27,182 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:27,182 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:27,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:27,182 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:27,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:27,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:27,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:27,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:27,183 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:57:27,186 DEBUG [Thread-1564] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:27,186 DEBUG [Thread-1564] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:27,281 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=941 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/B/73f655f32858409c9963dd40947c8de3 2023-03-20 10:57:27,286 DEBUG [Thread-1564] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:27,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:27,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:27,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:27,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:27,318 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:27,318 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:27,318 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:27,318 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:27,319 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:27,319 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:27,319 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:27,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:27,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:27,320 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:27,320 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:27,320 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,320 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:27,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:27,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:27,321 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:27,328 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:27,328 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:27,329 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:27,329 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:27,329 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:27,342 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:27,342 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:27,342 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=941 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/C/9ed9e7f6299044d9b88f30fcfd59890d 2023-03-20 10:57:27,342 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:27,342 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. started... 2023-03-20 10:57:27,342 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. as already flushing 2023-03-20 10:57:27,343 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:27,343 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:27,343 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:27,343 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,343 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:27,343 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:27,344 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:27,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:27,344 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:27,345 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:27,345 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:27,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:27,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:27,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:27,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:27,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:27,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2534af2d[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:27,347 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:27,347 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:27,347 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:27,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:27,348 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,348 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:27,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:27,348 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:27,348 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:27,348 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:27,348 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/A/6c802ff1c7604d9fb67fa2d19a4e97e5 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/6c802ff1c7604d9fb67fa2d19a4e97e5 2023-03-20 10:57:27,349 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:27,349 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:27,349 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:27,349 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:27,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:27,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:27,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:27,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:27,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:27,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:27,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:27,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:27,353 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:27,353 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:27,353 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:27,353 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:27,353 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:27,354 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:27,354 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:27,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:27,354 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:27,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:27,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:27,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:27,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:27,355 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:27,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:27,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:27,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:27,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:27,356 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/6c802ff1c7604d9fb67fa2d19a4e97e5, entries=200, sequenceid=941, filesize=14.3 K 2023-03-20 10:57:27,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:27,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:27,356 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,357 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/B/73f655f32858409c9963dd40947c8de3 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/73f655f32858409c9963dd40947c8de3 2023-03-20 10:57:27,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:27,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:27,360 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:27,360 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:27,360 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,360 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:27,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:27,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:27,360 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:27,360 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,360 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:27,360 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:27,360 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:27,360 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:57:27,360 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:27,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:27,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:27,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:27,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:27,361 DEBUG [Thread-1564] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:27,361 DEBUG [Thread-1564] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:27,367 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/73f655f32858409c9963dd40947c8de3, entries=150, sequenceid=941, filesize=11.9 K 2023-03-20 10:57:27,368 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/C/9ed9e7f6299044d9b88f30fcfd59890d as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/9ed9e7f6299044d9b88f30fcfd59890d 2023-03-20 10:57:27,373 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/9ed9e7f6299044d9b88f30fcfd59890d, entries=150, sequenceid=941, filesize=11.9 K 2023-03-20 10:57:27,375 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=60.38 KB/61830 for 72402f7d20e803d356627e208deb6fe2 in 583ms, sequenceid=941, compaction requested=true 2023-03-20 10:57:27,375 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 72402f7d20e803d356627e208deb6fe2: 2023-03-20 10:57:27,375 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:57:27,375 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-03-20 10:57:27,375 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:57:27,375 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-20 10:57:27,375 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-03-20 10:57:27,382 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 82756 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-03-20 10:57:27,382 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1912): 72402f7d20e803d356627e208deb6fe2/A is initiating minor compaction (all files) 2023-03-20 10:57:27,383 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 72402f7d20e803d356627e208deb6fe2/A in TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:27,383 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/e0f2074420494f18892dd06834b239df, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/edc884406ce34f9f9441ff0cd73956ba, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/0d344dfab12d46749d37c0a4e6b84600, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/4f8b9b806fb64895b4075ab6dfdf4928, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/807d4f3ae4514bb8b8b91ff8e52a7776, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/612f5f5d981543779183450ac9c6e51b, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/6c802ff1c7604d9fb67fa2d19a4e97e5] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp, totalSize=80.8 K 2023-03-20 10:57:27,384 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 80316 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-03-20 10:57:27,384 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): 72402f7d20e803d356627e208deb6fe2/B is initiating minor compaction (all files) 2023-03-20 10:57:27,384 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 72402f7d20e803d356627e208deb6fe2/B in TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:27,384 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/fc0188b7d1fa415fb217910da8214522, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/267a75d458d944a9a9304cef4a9881cb, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/bc6fd36293f34956a497b5a66c4f7507, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/5513e593024f49da943e0691c383825c, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/e967d0bedf4a47789400ecc5dd7c2ad2, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/ecbb1475a4fe40779070fd1a43948990, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/73f655f32858409c9963dd40947c8de3] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp, totalSize=78.4 K 2023-03-20 10:57:27,385 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting fc0188b7d1fa415fb217910da8214522, keycount=50, bloomtype=ROW, size=7.0 K, encoding=NONE, compression=NONE, seqNum=7, earliestPutTs=1679309839906 2023-03-20 10:57:27,385 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting e0f2074420494f18892dd06834b239df, keycount=50, bloomtype=ROW, size=7.0 K, encoding=NONE, compression=NONE, seqNum=7, earliestPutTs=1679309839906 2023-03-20 10:57:27,386 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 267a75d458d944a9a9304cef4a9881cb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=840, earliestPutTs=1679309841266 2023-03-20 10:57:27,386 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting edc884406ce34f9f9441ff0cd73956ba, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=840, earliestPutTs=1679309841266 2023-03-20 10:57:27,390 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting bc6fd36293f34956a497b5a66c4f7507, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=875, earliestPutTs=1679309841369 2023-03-20 10:57:27,391 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 0d344dfab12d46749d37c0a4e6b84600, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=875, earliestPutTs=1679309841369 2023-03-20 10:57:27,391 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 5513e593024f49da943e0691c383825c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=888, earliestPutTs=1679309842506 2023-03-20 10:57:27,391 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 4f8b9b806fb64895b4075ab6dfdf4928, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=888, earliestPutTs=1679309842506 2023-03-20 10:57:27,392 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting e967d0bedf4a47789400ecc5dd7c2ad2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=907, earliestPutTs=1679309843568 2023-03-20 10:57:27,392 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 807d4f3ae4514bb8b8b91ff8e52a7776, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=907, earliestPutTs=1679309843568 2023-03-20 10:57:27,393 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting ecbb1475a4fe40779070fd1a43948990, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=920, earliestPutTs=1679309844656 2023-03-20 10:57:27,393 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 73f655f32858409c9963dd40947c8de3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=941, earliestPutTs=1679309845766 2023-03-20 10:57:27,395 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 612f5f5d981543779183450ac9c6e51b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=920, earliestPutTs=1679309844656 2023-03-20 10:57:27,396 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 6c802ff1c7604d9fb67fa2d19a4e97e5, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=941, earliestPutTs=1679309845755 2023-03-20 10:57:27,426 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 72402f7d20e803d356627e208deb6fe2 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-20 10:57:27,427 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=A 2023-03-20 10:57:27,427 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:27,427 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=B 2023-03-20 10:57:27,427 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:27,427 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=C 2023-03-20 10:57:27,427 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:27,430 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] throttle.PressureAwareThroughputController(145): 72402f7d20e803d356627e208deb6fe2#A#compaction#213 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:57:27,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 72402f7d20e803d356627e208deb6fe2 2023-03-20 10:57:27,435 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): 72402f7d20e803d356627e208deb6fe2#B#compaction#214 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:57:27,461 DEBUG [Thread-1564] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:27,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:27,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:27,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:27,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:27,486 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:27,486 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:27,486 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:27,486 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:27,487 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:27,487 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:27,489 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:27,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:27,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:27,489 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:27,490 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,490 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:27,490 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:27,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:27,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:27,490 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:27,495 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:27,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55182 deadline: 1679309907488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,495 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:27,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55132 deadline: 1679309907488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,495 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:27,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55168 deadline: 1679309907491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,496 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:27,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55136 deadline: 1679309907494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,503 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:27,503 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:27,503 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:27,503 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:27,504 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:27,504 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:27,504 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:27,504 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:27,504 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. started... 2023-03-20 10:57:27,504 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. as already flushing 2023-03-20 10:57:27,504 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:27,504 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:27,504 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:27,504 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,504 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:27,504 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:27,505 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=954 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/A/626d86ecae6d40948450d611bc3cca4e 2023-03-20 10:57:27,505 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,506 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:27,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:27,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:27,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:27,506 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:27,506 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:27,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:27,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:27,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:27,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:27,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fa94524[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:27,508 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:27,508 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:27,509 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:27,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:27,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:27,509 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:27,509 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:27,509 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:27,509 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,510 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:27,510 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:27,510 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:27,510 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:27,511 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:27,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:27,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:27,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:27,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:27,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:27,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:27,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:27,517 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:27,517 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:27,517 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:27,517 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:27,517 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:27,517 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:27,519 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:27,519 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:27,519 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:27,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:27,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:27,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:27,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:27,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:27,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:27,519 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:27,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:27,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:27,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:27,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:27,521 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:27,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:27,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:27,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,524 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:27,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:27,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:27,524 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:27,524 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,524 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:27,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:27,524 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:27,524 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:27,524 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,524 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:27,524 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:27,524 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:27,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:27,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:27,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:27,524 DEBUG [Thread-1564] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:27,524 DEBUG [Thread-1564] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:27,525 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:57:27,555 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=954 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/B/a38aa04d1ad64cdb988405db45030ffc 2023-03-20 10:57:27,599 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:27,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55182 deadline: 1679309907596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,600 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:27,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55168 deadline: 1679309907597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,600 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:27,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55132 deadline: 1679309907597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,600 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:27,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55136 deadline: 1679309907597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,617 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=954 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/C/fef22325c3594bcab54e13d01f900e2c 2023-03-20 10:57:27,623 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/A/626d86ecae6d40948450d611bc3cca4e as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/626d86ecae6d40948450d611bc3cca4e 2023-03-20 10:57:27,625 DEBUG [Thread-1564] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:27,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:27,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:27,636 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/626d86ecae6d40948450d611bc3cca4e, entries=200, sequenceid=954, filesize=14.3 K 2023-03-20 10:57:27,638 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/B/a38aa04d1ad64cdb988405db45030ffc as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/a38aa04d1ad64cdb988405db45030ffc 2023-03-20 10:57:27,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:27,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:27,638 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:27,638 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:27,638 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:27,638 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:27,639 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:27,639 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:27,640 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,640 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:27,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:27,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:27,640 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,640 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:27,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:27,641 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:27,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:27,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:27,642 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:27,647 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:27,647 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:27,647 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:27,647 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:27,647 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:27,650 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:27,650 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:27,650 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:27,654 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. started... 2023-03-20 10:57:27,654 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. as already flushing 2023-03-20 10:57:27,654 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:27,654 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:27,654 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:27,655 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,655 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:27,655 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:27,656 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:27,657 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,657 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:27,657 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:27,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:27,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:27,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:27,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:27,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:27,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:27,660 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:27,660 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e06cc6f[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:27,660 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:27,660 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:27,662 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:27,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:27,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:27,662 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:27,662 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:27,662 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,662 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:27,662 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:27,664 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:27,664 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/a38aa04d1ad64cdb988405db45030ffc, entries=150, sequenceid=954, filesize=11.9 K 2023-03-20 10:57:27,664 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:27,664 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:27,664 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:27,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:27,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:27,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:27,665 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/C/fef22325c3594bcab54e13d01f900e2c as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/fef22325c3594bcab54e13d01f900e2c 2023-03-20 10:57:27,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:27,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:27,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:27,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:27,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:27,668 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:27,668 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:27,668 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:27,668 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:27,668 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:27,671 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:27,671 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:27,671 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:27,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:27,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:27,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:27,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:27,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:27,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:27,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:27,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:27,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:27,673 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:27,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:27,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:27,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,676 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:27,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:27,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,679 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/fef22325c3594bcab54e13d01f900e2c, entries=150, sequenceid=954, filesize=11.9 K 2023-03-20 10:57:27,683 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for 72402f7d20e803d356627e208deb6fe2 in 257ms, sequenceid=954, compaction requested=true 2023-03-20 10:57:27,683 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 72402f7d20e803d356627e208deb6fe2: 2023-03-20 10:57:27,683 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-20 10:57:27,683 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-03-20 10:57:27,683 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=4), splitQueue=0 2023-03-20 10:57:27,687 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:27,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:27,687 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:27,687 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:27,687 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:27,687 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,687 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:27,687 DEBUG [Thread-1564] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:27,687 DEBUG [Thread-1564] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:27,687 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:27,689 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:27,691 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:27,691 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:27,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:27,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:27,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:27,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:27,788 DEBUG [Thread-1564] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:27,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:27,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:27,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:27,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:27,791 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:27,791 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:27,791 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:27,791 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:27,793 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:27,793 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:27,794 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:27,794 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:27,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:27,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:27,795 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,795 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:27,795 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:27,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:27,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:27,795 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:27,795 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:27,795 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:27,796 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:27,796 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:27,796 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:27,796 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:27,796 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:27,796 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:27,796 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. started... 2023-03-20 10:57:27,796 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 72402f7d20e803d356627e208deb6fe2 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-20 10:57:27,796 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=A 2023-03-20 10:57:27,796 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:27,796 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=B 2023-03-20 10:57:27,797 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:27,797 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=C 2023-03-20 10:57:27,797 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:27,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 72402f7d20e803d356627e208deb6fe2 2023-03-20 10:57:27,803 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. as already flushing 2023-03-20 10:57:27,816 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=974 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/A/351daccce60f467a92e3f352b65ad8e1 2023-03-20 10:57:27,821 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:27,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55136 deadline: 1679309907813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,821 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:27,822 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:27,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55168 deadline: 1679309907813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55132 deadline: 1679309907820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,822 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:27,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55182 deadline: 1679309907821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,855 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=974 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/B/2a7041c802ed4d5d98d08ecc9ad0224d 2023-03-20 10:57:27,905 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/B/6e0a2c2ace774dafb0c2812476d863f2 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/6e0a2c2ace774dafb0c2812476d863f2 2023-03-20 10:57:27,905 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/A/7be9cb127c0a4694b69cf6619b74e3b7 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/7be9cb127c0a4694b69cf6619b74e3b7 2023-03-20 10:57:27,917 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 72402f7d20e803d356627e208deb6fe2/B of 72402f7d20e803d356627e208deb6fe2 into 6e0a2c2ace774dafb0c2812476d863f2(size=12.1 K), total size for store is 24.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:57:27,917 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 72402f7d20e803d356627e208deb6fe2: 2023-03-20 10:57:27,917 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2., storeName=72402f7d20e803d356627e208deb6fe2/B, priority=9, startTime=1679309847375; duration=0sec 2023-03-20 10:57:27,918 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=4), splitQueue=0 2023-03-20 10:57:27,918 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 7 compacting, 1 eligible, 16 blocking 2023-03-20 10:57:27,919 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-20 10:57:27,920 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-20 10:57:27,920 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. because compaction request was cancelled 2023-03-20 10:57:27,920 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-03-20 10:57:27,923 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 72402f7d20e803d356627e208deb6fe2/A of 72402f7d20e803d356627e208deb6fe2 into 7be9cb127c0a4694b69cf6619b74e3b7(size=12.1 K), total size for store is 26.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:57:27,923 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 72402f7d20e803d356627e208deb6fe2: 2023-03-20 10:57:27,923 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2., storeName=72402f7d20e803d356627e208deb6fe2/A, priority=9, startTime=1679309847375; duration=0sec 2023-03-20 10:57:27,923 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-20 10:57:27,924 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:27,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55118 deadline: 1679309907920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,925 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 92502 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-03-20 10:57:27,925 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): 72402f7d20e803d356627e208deb6fe2/C is initiating minor compaction (all files) 2023-03-20 10:57:27,925 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 72402f7d20e803d356627e208deb6fe2/C in TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:27,925 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/0de5c1a4660d403aaa2ca6c445f40991, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/f88cb61c68f646cbbf1eea0284a24779, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/40b1a0810c8e4ee3a04c8cdb6c180d49, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/40e5c692a5cd4e958aec77a681dbe2da, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/5a1be9933de24bc4acf224f5fd026dbd, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/e1a7d61b163b46ed8c107c37f9b782d8, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/9ed9e7f6299044d9b88f30fcfd59890d, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/fef22325c3594bcab54e13d01f900e2c] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp, totalSize=90.3 K 2023-03-20 10:57:27,926 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 8 compacting, 0 eligible, 16 blocking 2023-03-20 10:57:27,926 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:27,926 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-20 10:57:27,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55136 deadline: 1679309907923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,926 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-20 10:57:27,926 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:27,926 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 0de5c1a4660d403aaa2ca6c445f40991, keycount=50, bloomtype=ROW, size=7.0 K, encoding=NONE, compression=NONE, seqNum=7, earliestPutTs=1679309839906 2023-03-20 10:57:27,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55132 deadline: 1679309907923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,926 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. because compaction request was cancelled 2023-03-20 10:57:27,926 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-20 10:57:27,926 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:27,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55182 deadline: 1679309907923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,926 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting f88cb61c68f646cbbf1eea0284a24779, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=840, earliestPutTs=1679309841266 2023-03-20 10:57:27,926 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:27,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55168 deadline: 1679309907924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:27,927 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 40b1a0810c8e4ee3a04c8cdb6c180d49, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=875, earliestPutTs=1679309841369 2023-03-20 10:57:27,927 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-20 10:57:27,927 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-20 10:57:27,927 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. because compaction request was cancelled 2023-03-20 10:57:27,927 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 40e5c692a5cd4e958aec77a681dbe2da, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=888, earliestPutTs=1679309842506 2023-03-20 10:57:27,927 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 5a1be9933de24bc4acf224f5fd026dbd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=907, earliestPutTs=1679309843568 2023-03-20 10:57:27,928 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting e1a7d61b163b46ed8c107c37f9b782d8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=920, earliestPutTs=1679309844656 2023-03-20 10:57:27,928 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 9ed9e7f6299044d9b88f30fcfd59890d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=941, earliestPutTs=1679309845766 2023-03-20 10:57:27,928 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting fef22325c3594bcab54e13d01f900e2c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=954, earliestPutTs=1679309846800 2023-03-20 10:57:27,969 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): 72402f7d20e803d356627e208deb6fe2#C#compaction#221 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:57:27,993 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/C/a0883a66603149ebb7cf6a4b28270c60 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/a0883a66603149ebb7cf6a4b28270c60 2023-03-20 10:57:28,016 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 72402f7d20e803d356627e208deb6fe2/C of 72402f7d20e803d356627e208deb6fe2 into a0883a66603149ebb7cf6a4b28270c60(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:57:28,016 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 72402f7d20e803d356627e208deb6fe2: 2023-03-20 10:57:28,016 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2., storeName=72402f7d20e803d356627e208deb6fe2/C, priority=8, startTime=1679309847683; duration=0sec 2023-03-20 10:57:28,016 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:57:28,128 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:28,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55136 deadline: 1679309908127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,128 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:28,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55132 deadline: 1679309908127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,130 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:28,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55168 deadline: 1679309908127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,130 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:28,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55182 deadline: 1679309908128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,290 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=974 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/C/e18ee315a83546bb984a40b61e2a1158 2023-03-20 10:57:28,295 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/A/351daccce60f467a92e3f352b65ad8e1 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/351daccce60f467a92e3f352b65ad8e1 2023-03-20 10:57:28,300 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/351daccce60f467a92e3f352b65ad8e1, entries=150, sequenceid=974, filesize=11.9 K 2023-03-20 10:57:28,301 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/B/2a7041c802ed4d5d98d08ecc9ad0224d as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/2a7041c802ed4d5d98d08ecc9ad0224d 2023-03-20 10:57:28,307 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/2a7041c802ed4d5d98d08ecc9ad0224d, entries=150, sequenceid=974, filesize=11.9 K 2023-03-20 10:57:28,308 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/C/e18ee315a83546bb984a40b61e2a1158 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/e18ee315a83546bb984a40b61e2a1158 2023-03-20 10:57:28,315 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/e18ee315a83546bb984a40b61e2a1158, entries=150, sequenceid=974, filesize=11.9 K 2023-03-20 10:57:28,318 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 72402f7d20e803d356627e208deb6fe2 in 522ms, sequenceid=974, compaction requested=true 2023-03-20 10:57:28,319 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 72402f7d20e803d356627e208deb6fe2: 2023-03-20 10:57:28,319 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:28,319 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:28,319 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:28,319 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,322 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:28,322 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:28,323 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,323 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:28,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:28,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:28,323 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:28,323 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:28,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:28,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:28,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:28,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:28,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:28,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f19ddbb[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:28,326 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:28,326 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:28,327 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:28,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:28,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:28,327 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:28,327 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,327 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:28,327 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:28,327 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:28,329 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:28,329 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:28,329 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:28,329 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:28,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:28,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:28,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:28,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:28,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:28,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:28,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:28,332 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:28,332 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:28,332 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:28,332 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:28,332 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:28,332 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:28,334 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:28,334 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:28,334 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:28,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:28,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:28,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:28,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:28,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:28,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:28,334 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:28,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:28,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:28,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:28,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:28,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:28,336 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:28,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:28,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,338 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:28,338 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:28,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:28,339 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:28,339 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:28,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:28,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:28,339 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:28,339 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:28,339 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:28,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:28,339 DEBUG [Thread-1564] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:28,339 DEBUG [Thread-1564] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:28,339 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:28,339 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:28,339 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:28,339 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:57:28,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 72402f7d20e803d356627e208deb6fe2 2023-03-20 10:57:28,435 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 72402f7d20e803d356627e208deb6fe2 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-20 10:57:28,436 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=A 2023-03-20 10:57:28,436 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:28,436 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=B 2023-03-20 10:57:28,436 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:28,436 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=C 2023-03-20 10:57:28,436 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:28,439 DEBUG [Thread-1564] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:28,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:28,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:28,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:28,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:28,470 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:28,470 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:28,470 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:28,470 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:28,471 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:28,471 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:28,472 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,472 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,472 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:28,478 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:28,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:28,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:28,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:28,491 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:28,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:28,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:28,498 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:28,504 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:28,508 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:28,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55168 deadline: 1679309908497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,511 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:28,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55182 deadline: 1679309908502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,506 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:28,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55132 deadline: 1679309908496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,511 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:28,511 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:28,512 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:28,511 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:28,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55136 deadline: 1679309908502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,518 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:28,518 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:28,519 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:28,519 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. started... 2023-03-20 10:57:28,519 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. as already flushing 2023-03-20 10:57:28,519 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:28,519 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:28,519 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:28,519 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,519 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:28,519 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:28,534 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,534 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:28,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:28,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:28,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:28,535 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:28,535 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:28,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:28,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:28,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:28,536 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:28,536 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f5458b2[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:28,536 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:28,536 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:28,537 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:28,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:28,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:28,537 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:28,537 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:28,537 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:28,538 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,538 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:28,539 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:28,539 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:28,539 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:28,539 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:28,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:28,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:28,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:28,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:28,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:28,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:28,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:28,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:28,543 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:28,543 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:28,543 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:28,543 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:28,543 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:28,544 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:28,544 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:28,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:28,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:28,544 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:28,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:28,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:28,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:28,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:28,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:28,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:28,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:28,546 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:28,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:28,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:28,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,549 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:28,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:28,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,570 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:28,570 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:28,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:28,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:28,570 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,571 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:28,571 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:28,571 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,571 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:28,571 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:28,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:28,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:28,571 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:28,571 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:28,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:28,571 DEBUG [Thread-1564] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:28,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:28,571 DEBUG [Thread-1564] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:28,614 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:28,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55168 deadline: 1679309908612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,614 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:28,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55132 deadline: 1679309908612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,615 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:28,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55136 deadline: 1679309908613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,621 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:28,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55182 deadline: 1679309908618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,671 DEBUG [Thread-1564] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:28,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:28,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:28,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:28,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:28,677 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:28,677 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:28,677 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:28,677 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:28,677 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:28,677 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:28,678 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:28,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:28,678 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:28,679 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,679 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:28,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:28,679 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:28,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:28,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:28,680 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:28,680 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:28,680 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:28,680 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:28,680 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:28,680 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:28,680 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:28,681 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:28,681 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:28,681 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. started... 2023-03-20 10:57:28,681 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. as already flushing 2023-03-20 10:57:28,681 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:28,681 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:28,681 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:28,681 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,681 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:28,681 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:28,682 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:28,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:28,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:28,683 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:28,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:28,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:28,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:28,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:28,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ad1b219[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:28,686 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:28,686 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:28,690 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:28,690 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:28,695 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:28,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:28,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:28,696 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:28,696 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:28,696 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:28,696 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,696 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:28,705 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:28,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:28,706 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:28,706 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:28,706 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:28,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:28,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:28,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:28,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:28,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:28,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:28,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:28,712 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:28,712 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:28,712 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:28,712 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:28,712 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:28,714 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:28,714 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:28,714 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:28,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:28,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:28,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:28,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:28,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:28,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:28,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:28,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:28,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:28,716 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:28,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:28,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:28,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,717 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:28,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:28,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,721 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,721 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:28,721 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:28,721 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,721 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:28,721 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:28,721 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:28,721 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:28,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:28,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:28,721 DEBUG [Thread-1564] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:28,721 DEBUG [Thread-1564] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:28,722 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:28,722 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:28,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:28,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:28,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:28,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:28,818 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:28,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55168 deadline: 1679309908815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,820 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:28,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55132 deadline: 1679309908816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,820 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:28,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55136 deadline: 1679309908818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,822 DEBUG [Thread-1564] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:28,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:28,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:28,825 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:28,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55182 deadline: 1679309908825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:28,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:28,826 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:28,826 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:28,826 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:28,826 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:28,826 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:28,827 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:28,827 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:28,827 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:28,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:28,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:28,828 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,828 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:28,828 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:28,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:28,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:28,829 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:28,830 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:28,830 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:28,830 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:28,830 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:28,830 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:28,838 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:28,838 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:28,838 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:28,838 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. started... 2023-03-20 10:57:28,838 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. as already flushing 2023-03-20 10:57:28,838 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:28,838 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:28,838 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:28,838 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,838 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:28,838 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:28,839 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,839 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:28,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:28,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:28,840 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:28,840 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:28,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:28,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:28,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:28,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:28,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:28,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@428c6c98[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:28,841 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:28,841 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:28,842 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:28,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:28,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:28,842 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:28,842 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:28,842 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:28,842 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,842 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:28,843 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:28,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:28,843 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:28,843 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:28,843 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:28,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:28,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:28,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:28,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:28,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:28,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:28,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:28,845 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:28,845 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:28,845 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:28,845 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:28,845 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:28,846 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:28,846 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:28,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:28,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:28,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:28,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:28,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:28,846 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:28,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:28,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:28,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:28,847 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:28,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:28,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:28,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:28,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,849 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:28,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:28,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,850 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:28,851 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:28,851 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:28,851 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:28,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:28,851 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:28,851 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,851 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:28,851 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:28,851 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:28,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:28,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:28,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:28,851 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:28,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:28,851 DEBUG [Thread-1564] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:28,851 DEBUG [Thread-1564] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:28,884 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=991 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/A/0e3c87e90103476899411b351fe0c4d0 2023-03-20 10:57:28,913 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=991 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/B/9f154e8af2b64c4b945607fe41e00e82 2023-03-20 10:57:28,951 DEBUG [Thread-1564] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:28,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:28,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:28,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:28,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:28,965 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:28,965 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:28,965 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:28,965 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:28,965 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:28,965 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:28,966 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:28,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:28,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:28,966 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:28,967 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:28,967 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,967 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:28,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:28,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:28,968 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:28,970 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:28,970 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:28,970 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:28,970 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:28,970 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:28,986 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:28,986 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:28,986 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:28,986 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. started... 2023-03-20 10:57:28,986 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. as already flushing 2023-03-20 10:57:28,986 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:28,987 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:28,987 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:28,987 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,987 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:28,987 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:28,988 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,988 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:28,989 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:28,989 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:28,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:28,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:28,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:28,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:28,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:28,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:28,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:28,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73c99577[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:28,992 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:28,992 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:28,995 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:28,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:28,995 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,996 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:28,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:28,996 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:28,996 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:28,996 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:28,998 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,998 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:28,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:28,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:28,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:28,998 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:28,998 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:28,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:29,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:29,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:29,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:29,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:29,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:29,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:29,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:29,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:29,002 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:29,002 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:29,002 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:29,002 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:29,002 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:29,004 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:29,004 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:29,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:29,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:29,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:29,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:29,005 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:29,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:29,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:29,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:29,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:29,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:29,006 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:29,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:29,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:29,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:29,011 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:29,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:29,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:29,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:29,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:29,021 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:29,021 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:29,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:29,021 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:29,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:29,021 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:29,021 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:29,021 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:29,021 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:29,021 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:29,021 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:29,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:29,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:29,021 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:29,022 DEBUG [Thread-1564] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:29,022 DEBUG [Thread-1564] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:29,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:29,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:29,023 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:57:29,122 DEBUG [Thread-1564] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:29,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:29,123 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:29,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55168 deadline: 1679309909121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:29,124 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:29,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55132 deadline: 1679309909121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:29,124 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:29,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55136 deadline: 1679309909122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:29,125 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:29,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:29,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:29,126 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:29,126 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:29,126 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:29,126 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:29,127 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:29,127 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:29,128 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:29,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:29,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:29,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:29,129 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:29,129 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:29,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:29,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:29,130 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:29,130 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:29,130 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:29,131 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:29,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55182 deadline: 1679309909129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:29,142 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:29,142 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:29,144 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:29,144 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:29,144 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:29,150 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:29,150 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:29,150 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:29,150 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. started... 2023-03-20 10:57:29,150 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. as already flushing 2023-03-20 10:57:29,150 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:29,150 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:29,150 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:29,150 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:29,150 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:29,150 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:29,152 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:29,152 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:29,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:29,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:29,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:29,152 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:29,152 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:29,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:29,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:29,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:29,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:29,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:29,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:29,154 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:29,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e85d274[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:29,154 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:29,155 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:29,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:29,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:29,155 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:29,155 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:29,155 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:29,155 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:29,155 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:29,156 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:29,156 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:29,156 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:29,157 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:29,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:29,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:29,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:29,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:29,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:29,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:29,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:29,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:29,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:29,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:29,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:29,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:29,166 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:29,167 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:29,167 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:29,167 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:29,167 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:29,170 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:29,170 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:29,170 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:29,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:29,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:29,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:29,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:29,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:29,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:29,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:29,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:29,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:29,172 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:29,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:29,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:29,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:29,173 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:29,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:29,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:29,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:29,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:29,179 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:29,179 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:29,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:29,179 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:29,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:29,179 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:29,179 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:57:29,179 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:29,179 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:29,179 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:29,179 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:29,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:29,179 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:29,179 DEBUG [Thread-1564] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:29,179 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:29,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:29,179 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:29,179 DEBUG [Thread-1564] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:29,180 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:29,280 DEBUG [Thread-1564] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:29,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:29,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:29,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:29,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:29,283 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:29,283 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:29,283 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:29,283 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:29,283 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:29,283 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:29,284 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:29,285 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:29,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:29,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:29,285 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:29,285 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:29,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:29,285 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:29,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:29,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:29,286 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:29,286 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:29,286 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:29,286 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:29,286 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:29,286 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:29,286 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:29,286 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:29,286 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:29,287 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. started... 2023-03-20 10:57:29,287 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. as already flushing 2023-03-20 10:57:29,287 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:29,287 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:29,287 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:29,287 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:29,287 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:29,287 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:29,288 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:29,288 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:29,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:29,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:29,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:29,288 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:29,288 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:29,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:29,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:29,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:29,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:29,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:29,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:29,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5933ecc7[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:29,291 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:29,291 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:29,292 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:29,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:29,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:29,292 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:29,292 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:29,293 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:29,294 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:29,294 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:29,295 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:29,295 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:29,295 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:29,295 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:29,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:29,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:29,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:29,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:29,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:29,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:29,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:29,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:29,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:29,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:29,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:29,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:29,303 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:29,303 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:29,303 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:29,303 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:29,303 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:29,305 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:29,305 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:29,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:29,306 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:29,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:29,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:29,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:29,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:29,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:29,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:29,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:29,307 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:29,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:29,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:29,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:29,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:29,309 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:29,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:29,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:29,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:29,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:29,320 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:29,320 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:29,320 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:29,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:29,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:29,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:29,320 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:29,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:29,320 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:29,320 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:29,321 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:29,321 DEBUG [Thread-1564] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:29,321 DEBUG [Thread-1564] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:29,321 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:29,321 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:29,321 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:29,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:29,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:29,321 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:57:29,329 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=991 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/C/152e471c83f349038479f52c0c56df19 2023-03-20 10:57:29,338 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/A/0e3c87e90103476899411b351fe0c4d0 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/0e3c87e90103476899411b351fe0c4d0 2023-03-20 10:57:29,351 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/0e3c87e90103476899411b351fe0c4d0, entries=150, sequenceid=991, filesize=11.9 K 2023-03-20 10:57:29,355 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/B/9f154e8af2b64c4b945607fe41e00e82 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/9f154e8af2b64c4b945607fe41e00e82 2023-03-20 10:57:29,361 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/9f154e8af2b64c4b945607fe41e00e82, entries=150, sequenceid=991, filesize=11.9 K 2023-03-20 10:57:29,363 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/C/152e471c83f349038479f52c0c56df19 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/152e471c83f349038479f52c0c56df19 2023-03-20 10:57:29,384 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/152e471c83f349038479f52c0c56df19, entries=150, sequenceid=991, filesize=11.9 K 2023-03-20 10:57:29,384 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 72402f7d20e803d356627e208deb6fe2 in 949ms, sequenceid=991, compaction requested=true 2023-03-20 10:57:29,385 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 72402f7d20e803d356627e208deb6fe2: 2023-03-20 10:57:29,385 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 10:57:29,385 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:57:29,385 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:57:29,385 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-20 10:57:29,385 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 10:57:29,386 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51424 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 10:57:29,386 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1912): 72402f7d20e803d356627e208deb6fe2/A is initiating minor compaction (all files) 2023-03-20 10:57:29,387 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 72402f7d20e803d356627e208deb6fe2/A in TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:29,387 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/7be9cb127c0a4694b69cf6619b74e3b7, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/626d86ecae6d40948450d611bc3cca4e, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/351daccce60f467a92e3f352b65ad8e1, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/0e3c87e90103476899411b351fe0c4d0] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp, totalSize=50.2 K 2023-03-20 10:57:29,387 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48984 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 10:57:29,387 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 7be9cb127c0a4694b69cf6619b74e3b7, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=941, earliestPutTs=1679309845766 2023-03-20 10:57:29,387 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): 72402f7d20e803d356627e208deb6fe2/B is initiating minor compaction (all files) 2023-03-20 10:57:29,387 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 72402f7d20e803d356627e208deb6fe2/B in TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:29,387 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/6e0a2c2ace774dafb0c2812476d863f2, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/a38aa04d1ad64cdb988405db45030ffc, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/2a7041c802ed4d5d98d08ecc9ad0224d, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/9f154e8af2b64c4b945607fe41e00e82] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp, totalSize=47.8 K 2023-03-20 10:57:29,387 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 626d86ecae6d40948450d611bc3cca4e, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=954, earliestPutTs=1679309846800 2023-03-20 10:57:29,387 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 6e0a2c2ace774dafb0c2812476d863f2, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=941, earliestPutTs=1679309845766 2023-03-20 10:57:29,388 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting a38aa04d1ad64cdb988405db45030ffc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=954, earliestPutTs=1679309846800 2023-03-20 10:57:29,388 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 351daccce60f467a92e3f352b65ad8e1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=974, earliestPutTs=1679309847453 2023-03-20 10:57:29,388 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 2a7041c802ed4d5d98d08ecc9ad0224d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=974, earliestPutTs=1679309847453 2023-03-20 10:57:29,388 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 0e3c87e90103476899411b351fe0c4d0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=991, earliestPutTs=1679309847810 2023-03-20 10:57:29,389 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 9f154e8af2b64c4b945607fe41e00e82, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=991, earliestPutTs=1679309847810 2023-03-20 10:57:29,399 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] throttle.PressureAwareThroughputController(145): 72402f7d20e803d356627e208deb6fe2#A#compaction#225 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:57:29,406 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): 72402f7d20e803d356627e208deb6fe2#B#compaction#226 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:57:29,421 DEBUG [Thread-1564] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:29,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:29,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:29,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:29,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:29,446 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:29,446 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:29,446 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:29,446 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:29,447 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:29,447 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:29,448 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:29,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:29,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:29,448 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:29,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:29,449 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:29,449 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:29,449 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:29,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:29,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:29,450 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:29,456 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:29,456 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:29,456 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:29,456 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:29,456 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:29,457 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:29,457 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:29,457 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:29,457 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. started... 2023-03-20 10:57:29,457 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 72402f7d20e803d356627e208deb6fe2 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-20 10:57:29,458 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=A 2023-03-20 10:57:29,458 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:29,458 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=B 2023-03-20 10:57:29,458 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:29,458 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=C 2023-03-20 10:57:29,458 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:29,474 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/A/5cc3fd7208e04bfba831f54248749910 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/5cc3fd7208e04bfba831f54248749910 2023-03-20 10:57:29,481 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/B/f2926a57fb7942afb76bbe8e8db0b8d9 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/f2926a57fb7942afb76bbe8e8db0b8d9 2023-03-20 10:57:29,482 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=1009 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/A/82caf51f9cf24f05873667083f6ad1b2 2023-03-20 10:57:29,487 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 72402f7d20e803d356627e208deb6fe2/A of 72402f7d20e803d356627e208deb6fe2 into 5cc3fd7208e04bfba831f54248749910(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:57:29,487 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 72402f7d20e803d356627e208deb6fe2: 2023-03-20 10:57:29,487 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2., storeName=72402f7d20e803d356627e208deb6fe2/A, priority=12, startTime=1679309849385; duration=0sec 2023-03-20 10:57:29,487 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:57:29,487 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 10:57:29,488 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 72402f7d20e803d356627e208deb6fe2/B of 72402f7d20e803d356627e208deb6fe2 into f2926a57fb7942afb76bbe8e8db0b8d9(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:57:29,488 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 72402f7d20e803d356627e208deb6fe2: 2023-03-20 10:57:29,488 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2., storeName=72402f7d20e803d356627e208deb6fe2/B, priority=12, startTime=1679309849385; duration=0sec 2023-03-20 10:57:29,488 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:57:29,488 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36832 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 10:57:29,488 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1912): 72402f7d20e803d356627e208deb6fe2/C is initiating minor compaction (all files) 2023-03-20 10:57:29,488 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 72402f7d20e803d356627e208deb6fe2/C in TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:29,488 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/a0883a66603149ebb7cf6a4b28270c60, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/e18ee315a83546bb984a40b61e2a1158, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/152e471c83f349038479f52c0c56df19] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp, totalSize=36.0 K 2023-03-20 10:57:29,490 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting a0883a66603149ebb7cf6a4b28270c60, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=954, earliestPutTs=1679309846800 2023-03-20 10:57:29,492 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting e18ee315a83546bb984a40b61e2a1158, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=974, earliestPutTs=1679309847453 2023-03-20 10:57:29,492 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 152e471c83f349038479f52c0c56df19, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=991, earliestPutTs=1679309847810 2023-03-20 10:57:29,527 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] throttle.PressureAwareThroughputController(145): 72402f7d20e803d356627e208deb6fe2#C#compaction#229 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:57:29,581 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/C/612db19d8c00481fb117c2a2ec284186 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/612db19d8c00481fb117c2a2ec284186 2023-03-20 10:57:29,594 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 72402f7d20e803d356627e208deb6fe2/C of 72402f7d20e803d356627e208deb6fe2 into 612db19d8c00481fb117c2a2ec284186(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:57:29,594 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 72402f7d20e803d356627e208deb6fe2: 2023-03-20 10:57:29,594 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2., storeName=72402f7d20e803d356627e208deb6fe2/C, priority=13, startTime=1679309849385; duration=0sec 2023-03-20 10:57:29,594 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:57:29,630 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. as already flushing 2023-03-20 10:57:29,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 72402f7d20e803d356627e208deb6fe2 2023-03-20 10:57:29,647 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:29,647 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:29,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55168 deadline: 1679309909643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:29,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55182 deadline: 1679309909643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:29,655 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:29,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55136 deadline: 1679309909653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:29,656 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:29,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55132 deadline: 1679309909654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:29,750 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:29,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 287 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55168 deadline: 1679309909749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:29,752 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:29,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55182 deadline: 1679309909750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:29,758 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:29,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55136 deadline: 1679309909757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:29,758 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:29,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55132 deadline: 1679309909758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:29,944 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=1009 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/B/47dc95c30fae441aa87572d9b2ac4340 2023-03-20 10:57:29,948 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:29,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55118 deadline: 1679309909945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:29,958 DEBUG [Thread-1554] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4171 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2., hostname=jenkins-hbase20.apache.org,36041,1679309777062, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-20 10:57:29,961 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:29,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55182 deadline: 1679309909960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:29,961 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:29,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 289 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55168 deadline: 1679309909960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:29,962 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:29,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55132 deadline: 1679309909961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:29,966 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:29,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55136 deadline: 1679309909962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:29,994 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=1009 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/C/fbb612ed572d4ec78f167e43b8487fbe 2023-03-20 10:57:30,001 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/A/82caf51f9cf24f05873667083f6ad1b2 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/82caf51f9cf24f05873667083f6ad1b2 2023-03-20 10:57:30,028 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/82caf51f9cf24f05873667083f6ad1b2, entries=150, sequenceid=1009, filesize=11.9 K 2023-03-20 10:57:30,039 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/B/47dc95c30fae441aa87572d9b2ac4340 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/47dc95c30fae441aa87572d9b2ac4340 2023-03-20 10:57:30,049 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/47dc95c30fae441aa87572d9b2ac4340, entries=150, sequenceid=1009, filesize=11.9 K 2023-03-20 10:57:30,053 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/C/fbb612ed572d4ec78f167e43b8487fbe as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/fbb612ed572d4ec78f167e43b8487fbe 2023-03-20 10:57:30,086 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/fbb612ed572d4ec78f167e43b8487fbe, entries=150, sequenceid=1009, filesize=11.9 K 2023-03-20 10:57:30,088 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 72402f7d20e803d356627e208deb6fe2 in 631ms, sequenceid=1009, compaction requested=false 2023-03-20 10:57:30,088 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 72402f7d20e803d356627e208deb6fe2: 2023-03-20 10:57:30,088 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:30,088 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:30,089 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:30,089 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,089 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:30,089 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:30,090 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:30,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:30,090 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:30,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:30,091 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:30,091 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:30,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:30,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:30,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:30,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:30,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78fab3d5[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:30,092 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:30,092 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:30,094 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:30,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:30,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:30,095 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:30,095 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:30,095 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:30,097 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:30,097 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:30,097 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:30,098 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,099 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:30,099 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:30,099 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:30,099 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:30,099 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:30,099 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:30,099 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:30,102 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:30,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:30,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:30,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:30,103 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:30,103 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:30,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:30,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:30,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:30,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:30,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:30,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:30,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:30,105 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:30,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:30,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:30,107 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:30,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,113 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:30,113 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:30,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:30,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:30,113 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:30,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:30,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:30,114 DEBUG [Thread-1564] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:30,114 DEBUG [Thread-1564] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:30,118 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,118 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:30,118 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:30,118 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,118 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:30,118 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:30,118 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:30,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:30,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:30,214 DEBUG [Thread-1564] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:30,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:30,216 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:30,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:30,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:30,218 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:30,218 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:30,218 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:30,218 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:30,223 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:30,223 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:30,224 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,224 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:30,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:30,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:30,224 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,224 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:30,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:30,224 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:30,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:30,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:30,225 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:30,225 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:30,225 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:30,226 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:30,226 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:30,226 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:30,227 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:30,227 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:30,227 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:30,227 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. started... 2023-03-20 10:57:30,227 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 72402f7d20e803d356627e208deb6fe2 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-20 10:57:30,231 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=A 2023-03-20 10:57:30,232 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:30,232 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=B 2023-03-20 10:57:30,232 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:30,232 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=C 2023-03-20 10:57:30,232 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:30,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 72402f7d20e803d356627e208deb6fe2 2023-03-20 10:57:30,266 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. as already flushing 2023-03-20 10:57:30,296 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:30,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55182 deadline: 1679309910287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,298 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:30,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55132 deadline: 1679309910290, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,299 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:30,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 295 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55168 deadline: 1679309910295, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,308 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:30,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55136 deadline: 1679309910304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,315 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=1026 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/A/e2a8475cccf44a0c98fd75806af54c65 2023-03-20 10:57:30,402 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:30,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55132 deadline: 1679309910399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,406 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:30,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 297 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55168 deadline: 1679309910404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,411 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:30,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55182 deadline: 1679309910408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,413 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:30,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55136 deadline: 1679309910410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,424 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=1026 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/B/70cd43003c784875887fd459af37f552 2023-03-20 10:57:30,486 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=1026 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/C/d9a2b25c8fcb4182a6440333e3166d25 2023-03-20 10:57:30,501 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/A/e2a8475cccf44a0c98fd75806af54c65 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/e2a8475cccf44a0c98fd75806af54c65 2023-03-20 10:57:30,506 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/e2a8475cccf44a0c98fd75806af54c65, entries=150, sequenceid=1026, filesize=11.9 K 2023-03-20 10:57:30,507 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/B/70cd43003c784875887fd459af37f552 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/70cd43003c784875887fd459af37f552 2023-03-20 10:57:30,514 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/70cd43003c784875887fd459af37f552, entries=150, sequenceid=1026, filesize=11.9 K 2023-03-20 10:57:30,516 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/C/d9a2b25c8fcb4182a6440333e3166d25 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/d9a2b25c8fcb4182a6440333e3166d25 2023-03-20 10:57:30,548 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/d9a2b25c8fcb4182a6440333e3166d25, entries=150, sequenceid=1026, filesize=11.9 K 2023-03-20 10:57:30,549 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 72402f7d20e803d356627e208deb6fe2 in 322ms, sequenceid=1026, compaction requested=true 2023-03-20 10:57:30,549 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 72402f7d20e803d356627e208deb6fe2: 2023-03-20 10:57:30,549 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:30,549 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:30,549 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:30,549 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,549 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:30,550 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:30,550 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,550 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:30,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:30,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:30,551 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:30,551 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:30,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:30,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:30,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:30,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:30,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:30,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4727e390[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:30,553 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:30,553 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:30,554 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:30,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:30,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:30,554 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:30,554 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:30,554 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:30,554 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,555 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:30,555 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:30,555 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:30,555 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:30,556 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:30,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:30,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:30,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:30,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:30,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:30,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:30,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:30,559 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:30,559 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:30,559 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:30,559 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:30,559 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:30,559 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:30,561 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:30,561 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:30,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:30,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:30,561 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:30,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:30,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:30,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:30,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:30,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:30,563 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:30,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:30,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:30,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:30,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:30,565 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:30,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:30,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,566 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:30,567 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:30,567 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,567 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:30,567 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:30,567 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:57:30,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:30,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:30,567 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,567 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:30,568 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:30,568 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:30,567 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:30,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:30,567 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:30,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:30,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:30,568 DEBUG [Thread-1564] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:30,568 DEBUG [Thread-1564] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:30,615 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 72402f7d20e803d356627e208deb6fe2 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-20 10:57:30,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 72402f7d20e803d356627e208deb6fe2 2023-03-20 10:57:30,616 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=A 2023-03-20 10:57:30,617 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:30,617 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=B 2023-03-20 10:57:30,617 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:30,617 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=C 2023-03-20 10:57:30,617 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:30,646 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:30,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55132 deadline: 1679309910640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,646 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:30,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55168 deadline: 1679309910640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,647 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:30,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55136 deadline: 1679309910641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,651 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:30,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55182 deadline: 1679309910646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,668 DEBUG [Thread-1564] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:30,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:30,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:30,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:30,707 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=1046 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/A/7da10cda8de3491b9c7b236e9a248bd7 2023-03-20 10:57:30,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:30,707 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:30,707 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:30,707 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:30,707 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:30,707 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:30,707 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:30,708 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,708 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:30,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:30,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:30,709 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,709 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:30,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:30,709 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:30,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:30,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:30,710 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:30,714 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:30,714 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:30,714 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:30,714 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:30,714 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:30,722 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:30,722 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:30,722 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:30,722 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. started... 2023-03-20 10:57:30,722 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. as already flushing 2023-03-20 10:57:30,722 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:30,722 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:30,722 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:30,722 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,722 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:30,722 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:30,723 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:30,724 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:30,724 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:30,724 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:30,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:30,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:30,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:30,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:30,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:30,727 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:30,727 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47318f72[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:30,727 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:30,727 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:30,727 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:30,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:30,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:30,728 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:30,728 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:30,728 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:30,728 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,729 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:30,729 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:30,729 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:30,729 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:30,729 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:30,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:30,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:30,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:30,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:30,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:30,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:30,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:30,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:30,737 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:30,737 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:30,737 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:30,738 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:30,738 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:30,745 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:30,745 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:30,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:30,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:30,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:30,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:30,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:30,746 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,745 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:30,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:30,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:30,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:30,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:30,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:30,748 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:30,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:30,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:30,752 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:30,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55132 deadline: 1679309910747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,753 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:30,753 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:30,753 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:30,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55136 deadline: 1679309910748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,753 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:30,754 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:57:30,753 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,754 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:30,754 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:30,754 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,754 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:30,754 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:30,754 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:30,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:30,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:30,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:30,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:30,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:30,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:30,755 DEBUG [Thread-1564] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:30,755 DEBUG [Thread-1564] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:30,760 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:30,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55182 deadline: 1679309910754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,753 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:30,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 304 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55168 deadline: 1679309910748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,856 DEBUG [Thread-1564] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:30,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:30,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:30,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:30,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:30,870 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:30,870 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:30,870 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:30,870 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:30,871 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:30,871 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:30,873 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:30,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:30,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:30,873 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:30,875 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,875 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:30,875 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:30,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:30,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:30,876 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:30,878 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:30,878 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:30,878 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:30,878 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:30,878 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:30,882 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:30,882 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:30,883 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:30,884 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. started... 2023-03-20 10:57:30,884 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. as already flushing 2023-03-20 10:57:30,884 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:30,884 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:30,884 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:30,884 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,884 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:30,884 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:30,895 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:30,895 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:30,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:30,895 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:30,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:30,895 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:30,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:30,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:30,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:30,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:30,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1220687[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:30,897 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:30,897 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:30,898 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:30,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:30,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:30,899 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:30,899 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:30,899 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:30,903 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,903 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:30,905 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:30,905 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,905 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:30,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:30,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:30,905 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:30,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:30,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:30,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:30,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:30,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:30,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:30,907 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:30,907 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:30,907 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:30,907 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:30,907 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:30,907 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:30,908 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:30,908 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:30,908 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:30,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:30,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:30,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:30,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:30,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:30,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:30,909 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:30,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:30,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:30,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:30,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:30,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:30,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,910 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:30,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:30,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,913 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:30,913 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:30,913 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:30,913 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:30,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:30,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:30,913 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:30,913 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:30,913 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:30,913 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:30,913 DEBUG [Thread-1564] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:30,913 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:30,913 DEBUG [Thread-1564] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:30,914 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:30,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:30,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:30,958 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:30,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55132 deadline: 1679309910955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,959 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:30,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55136 deadline: 1679309910955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,964 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:30,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55182 deadline: 1679309910961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:30,971 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:30,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 306 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55168 deadline: 1679309910968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,014 DEBUG [Thread-1564] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:31,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:31,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:31,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:31,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:31,017 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:31,017 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:31,018 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:31,018 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:31,020 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:31,020 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:31,026 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:31,026 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,027 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,027 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:31,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:31,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:31,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:31,032 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:31,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:31,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:31,033 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:31,034 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:31,034 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:31,034 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:31,034 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:31,034 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:31,042 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:31,042 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:31,042 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:31,042 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. started... 2023-03-20 10:57:31,043 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. as already flushing 2023-03-20 10:57:31,043 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:31,043 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:31,043 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:31,043 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,043 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:31,043 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:31,045 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:31,046 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:31,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:31,046 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:31,046 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:31,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:31,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:31,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:31,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:31,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:31,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c6b796d[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:31,054 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:31,054 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:31,054 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:31,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:31,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:31,054 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:31,054 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:31,055 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:31,055 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,055 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:31,055 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,055 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:31,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:31,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:31,055 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:31,055 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:31,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:31,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:31,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:31,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:31,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:31,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:31,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:31,058 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:31,058 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:31,058 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:31,058 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:31,058 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:31,059 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:31,059 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:31,059 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:31,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:31,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:31,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:31,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:31,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:31,060 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:31,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:31,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:31,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:31,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:31,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:31,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:31,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,062 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:31,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:31,065 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:31,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:31,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:31,065 DEBUG [Thread-1564] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:31,065 DEBUG [Thread-1564] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:31,066 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:31,066 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:31,066 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,066 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:31,066 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:31,066 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,066 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:31,066 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:31,066 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:31,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:31,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:31,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:31,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:31,068 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:57:31,166 DEBUG [Thread-1564] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:31,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:31,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:31,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:31,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:31,170 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:31,170 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:31,170 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:31,170 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:31,170 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:31,170 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:31,171 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:31,171 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:31,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:31,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:31,172 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:31,172 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,172 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:31,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:31,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:31,173 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:31,178 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:31,179 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:31,179 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:31,179 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:31,179 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:31,179 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:31,179 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:31,179 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:31,179 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. started... 2023-03-20 10:57:31,179 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. as already flushing 2023-03-20 10:57:31,179 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:31,179 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:31,179 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:31,180 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,180 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:31,180 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:31,181 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:31,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:31,181 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:31,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:31,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:31,182 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:31,182 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:31,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:31,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:31,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:31,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54282413[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:31,183 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:31,183 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:31,184 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:31,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:31,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:31,185 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:31,185 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:31,185 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:31,185 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,185 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:31,186 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,186 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:31,186 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:31,186 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:31,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:31,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:31,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:31,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:31,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:31,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:31,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:31,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:31,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:31,190 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:31,190 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:31,190 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:31,190 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:31,190 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:31,192 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:31,192 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:31,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:31,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:31,192 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:31,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:31,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:31,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:31,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:31,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:31,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:31,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:31,193 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:31,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:31,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:31,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,194 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:31,194 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=1046 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/B/435b40fb08be42e4b58aa72f9ca6d775 2023-03-20 10:57:31,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:31,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,204 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:31,204 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:31,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:31,204 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:31,205 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:31,205 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:31,205 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,205 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:31,205 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:31,205 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:31,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:31,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:31,205 DEBUG [Thread-1564] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:31,205 DEBUG [Thread-1564] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:31,205 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:31,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:31,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:31,230 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=1046 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/C/cbe2d2a5562841edb86ec55ab4ddc298 2023-03-20 10:57:31,234 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/A/7da10cda8de3491b9c7b236e9a248bd7 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/7da10cda8de3491b9c7b236e9a248bd7 2023-03-20 10:57:31,239 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/7da10cda8de3491b9c7b236e9a248bd7, entries=200, sequenceid=1046, filesize=14.3 K 2023-03-20 10:57:31,240 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/B/435b40fb08be42e4b58aa72f9ca6d775 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/435b40fb08be42e4b58aa72f9ca6d775 2023-03-20 10:57:31,244 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/435b40fb08be42e4b58aa72f9ca6d775, entries=150, sequenceid=1046, filesize=11.9 K 2023-03-20 10:57:31,245 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/C/cbe2d2a5562841edb86ec55ab4ddc298 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/cbe2d2a5562841edb86ec55ab4ddc298 2023-03-20 10:57:31,248 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/cbe2d2a5562841edb86ec55ab4ddc298, entries=150, sequenceid=1046, filesize=11.9 K 2023-03-20 10:57:31,249 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 72402f7d20e803d356627e208deb6fe2 in 634ms, sequenceid=1046, compaction requested=true 2023-03-20 10:57:31,249 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 72402f7d20e803d356627e208deb6fe2: 2023-03-20 10:57:31,249 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:57:31,249 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 10:57:31,249 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 10:57:31,249 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:57:31,250 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:57:31,251 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51560 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 10:57:31,251 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1912): 72402f7d20e803d356627e208deb6fe2/A is initiating minor compaction (all files) 2023-03-20 10:57:31,251 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 72402f7d20e803d356627e208deb6fe2/A in TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:31,251 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/5cc3fd7208e04bfba831f54248749910, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/82caf51f9cf24f05873667083f6ad1b2, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/e2a8475cccf44a0c98fd75806af54c65, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/7da10cda8de3491b9c7b236e9a248bd7] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp, totalSize=50.4 K 2023-03-20 10:57:31,251 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49120 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 10:57:31,251 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): 72402f7d20e803d356627e208deb6fe2/B is initiating minor compaction (all files) 2023-03-20 10:57:31,251 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 72402f7d20e803d356627e208deb6fe2/B in TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:31,251 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/f2926a57fb7942afb76bbe8e8db0b8d9, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/47dc95c30fae441aa87572d9b2ac4340, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/70cd43003c784875887fd459af37f552, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/435b40fb08be42e4b58aa72f9ca6d775] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp, totalSize=48.0 K 2023-03-20 10:57:31,252 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 5cc3fd7208e04bfba831f54248749910, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=991, earliestPutTs=1679309847810 2023-03-20 10:57:31,252 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting f2926a57fb7942afb76bbe8e8db0b8d9, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=991, earliestPutTs=1679309847810 2023-03-20 10:57:31,252 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 82caf51f9cf24f05873667083f6ad1b2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1009, earliestPutTs=1679309848484 2023-03-20 10:57:31,252 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 47dc95c30fae441aa87572d9b2ac4340, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1009, earliestPutTs=1679309848484 2023-03-20 10:57:31,252 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 70cd43003c784875887fd459af37f552, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1026, earliestPutTs=1679309849637 2023-03-20 10:57:31,252 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting e2a8475cccf44a0c98fd75806af54c65, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1026, earliestPutTs=1679309849637 2023-03-20 10:57:31,253 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 435b40fb08be42e4b58aa72f9ca6d775, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1046, earliestPutTs=1679309850293 2023-03-20 10:57:31,253 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 7da10cda8de3491b9c7b236e9a248bd7, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=1046, earliestPutTs=1679309850293 2023-03-20 10:57:31,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 72402f7d20e803d356627e208deb6fe2 2023-03-20 10:57:31,264 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 72402f7d20e803d356627e208deb6fe2 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-20 10:57:31,264 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=A 2023-03-20 10:57:31,265 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:31,265 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=B 2023-03-20 10:57:31,265 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:31,265 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=C 2023-03-20 10:57:31,265 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:31,281 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): 72402f7d20e803d356627e208deb6fe2#B#compaction#238 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:57:31,284 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] throttle.PressureAwareThroughputController(145): 72402f7d20e803d356627e208deb6fe2#A#compaction#239 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:57:31,301 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:31,301 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:31,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 311 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55168 deadline: 1679309911292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55136 deadline: 1679309911292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,305 DEBUG [Thread-1564] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:31,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:31,307 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:31,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55182 deadline: 1679309911302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,307 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:31,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55132 deadline: 1679309911302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:31,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:31,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:31,354 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:31,354 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:31,354 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:31,354 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:31,355 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:31,355 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:31,355 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,355 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:31,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:31,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:31,356 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,356 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:31,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:31,356 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:31,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:31,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:31,357 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:31,362 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:31,362 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:31,362 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:31,362 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:31,362 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:31,362 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:31,362 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:31,363 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:31,363 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. started... 2023-03-20 10:57:31,363 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. as already flushing 2023-03-20 10:57:31,363 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:31,363 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:31,363 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:31,363 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,363 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:31,363 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:31,365 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/B/a379fa2d36f44455be9902dc6e1d3e46 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/a379fa2d36f44455be9902dc6e1d3e46 2023-03-20 10:57:31,370 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 72402f7d20e803d356627e208deb6fe2/B of 72402f7d20e803d356627e208deb6fe2 into a379fa2d36f44455be9902dc6e1d3e46(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:57:31,370 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 72402f7d20e803d356627e208deb6fe2: 2023-03-20 10:57:31,370 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2., storeName=72402f7d20e803d356627e208deb6fe2/B, priority=12, startTime=1679309851249; duration=0sec 2023-03-20 10:57:31,370 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:57:31,370 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 10:57:31,370 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:31,372 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49120 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 10:57:31,372 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): 72402f7d20e803d356627e208deb6fe2/C is initiating minor compaction (all files) 2023-03-20 10:57:31,372 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 72402f7d20e803d356627e208deb6fe2/C in TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:31,372 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/612db19d8c00481fb117c2a2ec284186, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/fbb612ed572d4ec78f167e43b8487fbe, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/d9a2b25c8fcb4182a6440333e3166d25, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/cbe2d2a5562841edb86ec55ab4ddc298] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp, totalSize=48.0 K 2023-03-20 10:57:31,373 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 612db19d8c00481fb117c2a2ec284186, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=991, earliestPutTs=1679309847810 2023-03-20 10:57:31,373 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting fbb612ed572d4ec78f167e43b8487fbe, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1009, earliestPutTs=1679309848484 2023-03-20 10:57:31,373 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting d9a2b25c8fcb4182a6440333e3166d25, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1026, earliestPutTs=1679309849637 2023-03-20 10:57:31,373 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting cbe2d2a5562841edb86ec55ab4ddc298, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1046, earliestPutTs=1679309850293 2023-03-20 10:57:31,374 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:31,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:31,374 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:31,375 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:31,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:31,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:31,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:31,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,380 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): 72402f7d20e803d356627e208deb6fe2#C#compaction#240 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:57:31,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:31,382 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:31,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1fa97bfb[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:31,383 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:31,383 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:31,384 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:31,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:31,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:31,384 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:31,384 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:31,384 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:31,384 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,384 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:31,385 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:31,385 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:31,385 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:31,388 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:31,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:31,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:31,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:31,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:31,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:31,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:31,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:31,394 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:31,394 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:31,394 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:31,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:31,394 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:31,395 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:31,396 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:31,396 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:31,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:31,396 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:31,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:31,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:31,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:31,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:31,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:31,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:31,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:31,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:31,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:31,397 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:31,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:31,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,398 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:31,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:31,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,402 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:31,402 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:31,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:31,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:31,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:31,402 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:31,402 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:31,403 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,403 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:31,403 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:31,403 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:31,403 DEBUG [Thread-1564] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:31,403 DEBUG [Thread-1564] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:31,402 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:31,402 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:31,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:31,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:31,403 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:57:31,403 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:31,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 313 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55168 deadline: 1679309911403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,404 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:31,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55136 deadline: 1679309911403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,409 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/C/dc5782915282432d9cc78dda1b2a9b45 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/dc5782915282432d9cc78dda1b2a9b45 2023-03-20 10:57:31,412 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:31,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55182 deadline: 1679309911408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,412 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:31,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55132 deadline: 1679309911408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,415 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 72402f7d20e803d356627e208deb6fe2/C of 72402f7d20e803d356627e208deb6fe2 into dc5782915282432d9cc78dda1b2a9b45(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:57:31,415 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 72402f7d20e803d356627e208deb6fe2: 2023-03-20 10:57:31,415 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2., storeName=72402f7d20e803d356627e208deb6fe2/C, priority=12, startTime=1679309851250; duration=0sec 2023-03-20 10:57:31,415 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:57:31,503 DEBUG [Thread-1564] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:31,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:31,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:31,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:31,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:31,514 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:31,514 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:31,514 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:31,514 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:31,515 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:31,515 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:31,515 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:31,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:31,515 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:31,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:31,516 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,516 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:31,516 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:31,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:31,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:31,517 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:31,517 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:31,517 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:31,517 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:31,517 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:31,517 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:31,518 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:31,518 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:31,518 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:31,518 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. started... 2023-03-20 10:57:31,518 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. as already flushing 2023-03-20 10:57:31,518 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:31,518 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:31,518 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:31,518 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,518 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:31,518 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:31,520 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:31,520 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:31,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:31,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:31,520 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:31,520 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:31,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:31,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:31,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:31,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:31,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@eae40b7[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:31,523 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:31,523 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:31,523 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:31,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:31,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:31,523 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:31,523 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:31,524 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:31,524 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,524 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:31,525 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:31,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:31,525 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:31,525 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:31,525 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:31,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:31,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:31,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:31,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:31,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:31,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:31,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:31,527 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:31,527 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:31,527 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:31,527 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:31,527 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:31,528 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:31,528 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:31,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:31,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:31,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:31,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:31,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:31,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:31,528 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:31,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:31,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:31,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:31,529 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:31,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:31,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:31,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,530 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:31,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:31,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,535 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:31,536 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:31,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:31,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:31,535 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,535 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:31,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:31,536 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:31,536 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:31,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:31,536 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,536 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:31,536 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:31,536 DEBUG [Thread-1564] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:31,536 DEBUG [Thread-1564] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:31,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:31,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:31,536 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:31,536 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:57:31,608 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:31,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55136 deadline: 1679309911606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,608 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:31,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 315 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55168 deadline: 1679309911607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,616 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:31,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55132 deadline: 1679309911613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,616 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:31,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55182 deadline: 1679309911614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,636 DEBUG [Thread-1564] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:31,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:31,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:31,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:31,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:31,640 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:31,640 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:31,640 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:31,640 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:31,641 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:31,641 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:31,642 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:31,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:31,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:31,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:31,642 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,643 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,643 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:31,643 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:31,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:31,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:31,644 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:31,645 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:31,645 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:31,645 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:31,645 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:31,645 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:31,645 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:31,645 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:31,645 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:31,645 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. started... 2023-03-20 10:57:31,645 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. as already flushing 2023-03-20 10:57:31,645 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:31,645 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:31,645 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:31,645 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,645 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:31,645 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:31,653 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,654 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:31,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:31,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:31,654 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:31,654 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:31,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:31,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:31,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:31,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:31,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:31,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7979e01e[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:31,657 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:31,657 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:31,659 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:31,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:31,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:31,659 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:31,659 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:31,659 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:31,660 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,660 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:31,660 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:31,660 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:31,660 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:31,660 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:31,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:31,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:31,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:31,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:31,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:31,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:31,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:31,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:31,664 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:31,664 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:31,664 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:31,664 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:31,664 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:31,665 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:31,665 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:31,665 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:31,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:31,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:31,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:31,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:31,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:31,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:31,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:31,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:31,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:31,667 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:31,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:31,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:31,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,669 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:31,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:31,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,671 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,671 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:31,671 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:31,671 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,671 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:31,671 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:31,671 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:31,671 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:31,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:31,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:31,672 DEBUG [Thread-1564] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:31,672 DEBUG [Thread-1564] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:31,672 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:31,672 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:31,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:31,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:31,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:31,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:31,728 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=1060 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/A/b146f4acfc4f4bcd928a9ed18b614e17 2023-03-20 10:57:31,760 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/A/bcc1226eb9f1486b9a920fd091ec88c6 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/bcc1226eb9f1486b9a920fd091ec88c6 2023-03-20 10:57:31,769 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 72402f7d20e803d356627e208deb6fe2/A of 72402f7d20e803d356627e208deb6fe2 into bcc1226eb9f1486b9a920fd091ec88c6(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:57:31,769 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 72402f7d20e803d356627e208deb6fe2: 2023-03-20 10:57:31,769 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2., storeName=72402f7d20e803d356627e208deb6fe2/A, priority=12, startTime=1679309851249; duration=0sec 2023-03-20 10:57:31,769 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:57:31,772 DEBUG [Thread-1564] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:31,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:31,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:31,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:31,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:31,777 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:31,777 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:31,777 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:31,777 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:31,780 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:31,780 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:31,787 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,790 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:31,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:31,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:31,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:31,795 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,795 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:31,798 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:31,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:31,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:31,802 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:31,806 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:31,806 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:31,806 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:31,806 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:31,806 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:31,814 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:31,814 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:31,814 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:31,814 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. started... 2023-03-20 10:57:31,814 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. as already flushing 2023-03-20 10:57:31,814 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:31,814 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:31,815 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:31,815 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,815 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:31,815 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:31,822 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:31,822 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:31,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:31,823 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:31,823 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:31,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:31,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:31,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:31,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:31,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:31,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c644b9e[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:31,832 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:31,832 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:31,837 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,837 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:31,842 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:31,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:31,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:31,842 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:31,842 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:31,842 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:31,845 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,845 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:31,845 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:31,845 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:31,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:31,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:31,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:31,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:31,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:31,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:31,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:31,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:31,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:31,853 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:31,853 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:31,853 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:31,853 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:31,853 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:31,855 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:31,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:31,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:31,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:31,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:31,855 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:31,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:31,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:31,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,857 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:31,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:31,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,859 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,859 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:31,859 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:31,859 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:31,859 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:31,859 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:31,859 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,860 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:31,860 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:31,860 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:31,860 DEBUG [Thread-1564] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:31,860 DEBUG [Thread-1564] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:31,862 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:31,862 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:31,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:31,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:31,863 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:31,866 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:57:31,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:31,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:31,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:31,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:31,912 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:31,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 317 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55168 deadline: 1679309911910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,913 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:31,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55136 deadline: 1679309911911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,920 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:31,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55132 deadline: 1679309911919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,921 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:31,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55182 deadline: 1679309911919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,960 DEBUG [Thread-1564] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:31,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:31,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:31,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:31,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:31,964 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:31,964 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:31,964 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:31,964 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:31,964 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:31,964 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:31,965 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:31,965 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:31,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:31,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:31,965 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,965 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:31,966 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:31,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:31,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:31,966 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:31,966 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:31,966 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:31,966 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:31,966 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:31,966 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:31,967 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:31,967 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:31,967 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:31,967 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. started... 2023-03-20 10:57:31,967 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. as already flushing 2023-03-20 10:57:31,967 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:31,967 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:31,967 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:31,967 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,967 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:31,967 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:31,975 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:31,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:31,975 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:31,976 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:31,976 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:31,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:31,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:31,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:31,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:31,978 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:31,978 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38c6834a[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:31,978 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:31,978 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:31,979 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:31,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:31,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:31,979 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:31,979 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:31,979 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:31,979 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,979 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:31,982 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:31,982 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:31,982 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:31,982 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:31,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:31,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:31,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:31,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:31,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:31,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:31,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:31,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:31,985 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:31,986 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:31,986 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:31,986 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:31,986 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:31,987 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:31,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:31,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:31,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:31,988 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:31,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:31,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:31,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:31,989 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,989 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:31,989 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:31,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:31,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:31,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:31,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:31,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:31,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:31,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:31,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,993 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:31,993 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:31,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:31,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:31,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:31,993 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:31,993 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:31,993 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:31,993 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:31,993 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:31,993 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:31,993 DEBUG [Thread-1564] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:31,993 DEBUG [Thread-1564] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:31,993 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:31,993 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:31,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:31,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:31,994 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:57:32,093 DEBUG [Thread-1564] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:32,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:32,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:32,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:32,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:32,097 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:32,097 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:32,098 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:32,098 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:32,098 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:32,098 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:32,099 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,099 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:32,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:32,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:32,099 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,099 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:32,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:32,099 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:32,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:32,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:32,100 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:32,100 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:32,100 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:32,100 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:32,100 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:32,100 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:32,100 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:32,100 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:32,100 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:32,100 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. started... 2023-03-20 10:57:32,100 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. as already flushing 2023-03-20 10:57:32,100 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:32,100 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:32,101 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:32,101 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,101 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:32,101 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:32,101 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,101 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:32,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:32,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:32,102 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:32,102 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:32,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:32,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:32,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:32,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:32,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:32,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4491d71d[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:32,103 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:32,104 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:32,105 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:32,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:32,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:32,105 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:32,105 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:32,105 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:32,105 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,105 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:32,106 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,106 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:32,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:32,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:32,106 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:32,106 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:32,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:32,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:32,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:32,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:32,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:32,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:32,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:32,108 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:32,108 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:32,108 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:32,108 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:32,108 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:32,109 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:32,109 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:32,109 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:32,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:32,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:32,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:32,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:32,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:32,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:32,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:32,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:32,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:32,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:32,110 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:32,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:32,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,111 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:32,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:32,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,114 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:32,114 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:32,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:32,114 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:32,114 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:32,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:32,114 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:32,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:32,114 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:32,114 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,114 DEBUG [Thread-1564] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:32,114 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:32,115 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:32,114 DEBUG [Thread-1564] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:32,115 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:32,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:32,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:32,176 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=1060 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/B/b0145fca1319491bbf81eb70b6dddfde 2023-03-20 10:57:32,198 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=1060 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/C/1d3848cec2e541259d2816cf2fd28d80 2023-03-20 10:57:32,212 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/A/b146f4acfc4f4bcd928a9ed18b614e17 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/b146f4acfc4f4bcd928a9ed18b614e17 2023-03-20 10:57:32,215 DEBUG [Thread-1564] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:32,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:32,217 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/b146f4acfc4f4bcd928a9ed18b614e17, entries=200, sequenceid=1060, filesize=14.3 K 2023-03-20 10:57:32,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:32,218 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/B/b0145fca1319491bbf81eb70b6dddfde as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/b0145fca1319491bbf81eb70b6dddfde 2023-03-20 10:57:32,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:32,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:32,220 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:32,220 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:32,220 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:32,220 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:32,222 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:32,222 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:32,224 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:32,224 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:32,225 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/b0145fca1319491bbf81eb70b6dddfde, entries=150, sequenceid=1060, filesize=11.9 K 2023-03-20 10:57:32,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:32,225 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/C/1d3848cec2e541259d2816cf2fd28d80 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/1d3848cec2e541259d2816cf2fd28d80 2023-03-20 10:57:32,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:32,226 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,226 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:32,226 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:32,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:32,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:32,227 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:32,227 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:32,228 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:32,228 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:32,228 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:32,228 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:32,228 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:32,228 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:32,228 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:32,228 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. started... 2023-03-20 10:57:32,228 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. as already flushing 2023-03-20 10:57:32,228 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:32,228 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:32,228 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:32,228 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,228 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:32,228 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:32,229 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,229 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:32,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:32,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:32,230 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:32,230 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:32,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:32,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:32,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:32,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:32,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:32,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72dea681[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:32,232 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/1d3848cec2e541259d2816cf2fd28d80, entries=150, sequenceid=1060, filesize=11.9 K 2023-03-20 10:57:32,232 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:32,232 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:32,233 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:32,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:32,233 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for 72402f7d20e803d356627e208deb6fe2 in 970ms, sequenceid=1060, compaction requested=false 2023-03-20 10:57:32,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:32,233 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 72402f7d20e803d356627e208deb6fe2: 2023-03-20 10:57:32,233 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:32,233 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:32,233 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:32,233 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,233 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:32,234 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:32,234 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:32,234 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:32,234 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:32,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:32,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:32,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:32,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:32,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:32,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:32,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:32,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:32,237 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:32,237 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:32,237 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:32,237 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:32,237 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:32,239 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:32,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:32,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:32,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:32,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:32,239 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:32,239 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:32,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:32,240 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:32,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:32,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:32,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:32,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:32,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:32,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:32,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,243 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:32,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:32,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,254 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:32,254 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:32,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:32,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:32,255 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,255 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:32,255 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:32,255 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,255 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:32,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:32,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:32,255 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:32,255 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:32,255 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:32,255 DEBUG [Thread-1564] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:32,255 DEBUG [Thread-1564] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:32,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:32,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:32,355 DEBUG [Thread-1564] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:32,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:32,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:32,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:32,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:32,372 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:32,372 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:32,372 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:32,372 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:32,373 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:32,373 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:32,373 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:32,374 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:32,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:32,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:32,375 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:32,375 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,375 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:32,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:32,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:32,376 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:32,386 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:32,386 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:32,386 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:32,386 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:32,386 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:32,386 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:32,386 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:32,386 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:32,387 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. started... 2023-03-20 10:57:32,387 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 72402f7d20e803d356627e208deb6fe2 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-20 10:57:32,387 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=A 2023-03-20 10:57:32,387 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:32,387 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=B 2023-03-20 10:57:32,387 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:32,388 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=C 2023-03-20 10:57:32,388 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:32,418 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=1082 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/A/f9456bb1ce61475f84976af7da9a70f1 2023-03-20 10:57:32,418 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. as already flushing 2023-03-20 10:57:32,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 72402f7d20e803d356627e208deb6fe2 2023-03-20 10:57:32,433 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:32,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 323 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55168 deadline: 1679309912429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,433 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:32,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55182 deadline: 1679309912431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,438 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:32,438 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:32,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55136 deadline: 1679309912434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55132 deadline: 1679309912434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,439 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=1082 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/B/0ca593230ae54512968e6a7b5bf2c0b9 2023-03-20 10:57:32,461 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=1082 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/C/8d010a3e3b4a4a43bba443e353262666 2023-03-20 10:57:32,469 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/A/f9456bb1ce61475f84976af7da9a70f1 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/f9456bb1ce61475f84976af7da9a70f1 2023-03-20 10:57:32,474 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/f9456bb1ce61475f84976af7da9a70f1, entries=150, sequenceid=1082, filesize=11.9 K 2023-03-20 10:57:32,475 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/B/0ca593230ae54512968e6a7b5bf2c0b9 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/0ca593230ae54512968e6a7b5bf2c0b9 2023-03-20 10:57:32,479 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/0ca593230ae54512968e6a7b5bf2c0b9, entries=150, sequenceid=1082, filesize=11.9 K 2023-03-20 10:57:32,480 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/C/8d010a3e3b4a4a43bba443e353262666 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/8d010a3e3b4a4a43bba443e353262666 2023-03-20 10:57:32,486 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/8d010a3e3b4a4a43bba443e353262666, entries=150, sequenceid=1082, filesize=11.9 K 2023-03-20 10:57:32,487 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 72402f7d20e803d356627e208deb6fe2 in 100ms, sequenceid=1082, compaction requested=true 2023-03-20 10:57:32,487 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 72402f7d20e803d356627e208deb6fe2: 2023-03-20 10:57:32,487 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:32,487 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:32,487 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:32,487 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,487 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:32,488 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:32,489 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:32,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:32,489 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:32,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:32,491 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:32,491 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:32,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:32,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:32,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:32,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:32,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c283bff[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:32,496 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:32,496 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:32,505 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:32,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:32,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:32,505 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:32,505 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:32,505 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:32,505 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,505 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:32,506 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:32,506 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:32,506 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:32,506 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:32,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:32,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:32,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:32,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:32,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:32,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:32,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:32,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:32,509 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:32,509 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:32,509 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:32,509 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:32,509 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:32,510 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:32,510 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:32,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:32,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:32,510 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:32,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:32,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:32,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:32,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:32,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:32,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:32,511 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:32,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:32,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:32,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:32,513 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:32,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:32,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,517 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:32,517 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:32,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:32,517 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:57:32,517 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:32,518 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:32,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:32,518 DEBUG [Thread-1564] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:32,518 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:32,518 DEBUG [Thread-1564] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:32,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:32,518 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:32,518 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,518 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:32,518 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:32,518 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:32,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:32,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:32,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 72402f7d20e803d356627e208deb6fe2 2023-03-20 10:57:32,537 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 72402f7d20e803d356627e208deb6fe2 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-20 10:57:32,537 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=A 2023-03-20 10:57:32,537 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:32,537 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=B 2023-03-20 10:57:32,537 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:32,537 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=C 2023-03-20 10:57:32,537 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:32,547 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=1097 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/A/24c544bc9cf942a08b5f9c6fe7d59bc6 2023-03-20 10:57:32,580 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=1097 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/B/cf227877aeac411bbfd455262e019b35 2023-03-20 10:57:32,592 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:32,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 294 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55182 deadline: 1679309912586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,592 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:32,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55136 deadline: 1679309912587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,592 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:32,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 330 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55168 deadline: 1679309912588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,592 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:32,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55132 deadline: 1679309912589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,618 DEBUG [Thread-1564] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:32,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:32,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:32,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:32,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:32,626 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:32,626 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:32,626 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:32,626 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:32,627 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:32,627 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:32,627 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:32,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:32,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:32,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:32,628 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,628 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,628 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:32,628 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:32,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:32,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:32,629 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:32,648 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:32,648 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:32,648 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=1097 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/C/9fd8492d5bdd4efb8d9d5eed33aeac40 2023-03-20 10:57:32,648 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:32,648 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:32,648 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:32,649 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:32,649 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:32,649 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:32,649 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. started... 2023-03-20 10:57:32,649 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. as already flushing 2023-03-20 10:57:32,649 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:32,649 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:32,649 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:32,649 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,649 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:32,649 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:32,651 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,651 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:32,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:32,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:32,652 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:32,652 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:32,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:32,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:32,653 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/A/24c544bc9cf942a08b5f9c6fe7d59bc6 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/24c544bc9cf942a08b5f9c6fe7d59bc6 2023-03-20 10:57:32,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:32,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:32,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:32,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30f833e3[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:32,654 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:32,654 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:32,655 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:32,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:32,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:32,656 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:32,656 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:32,656 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:32,657 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:32,657 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:32,657 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:32,657 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,658 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/24c544bc9cf942a08b5f9c6fe7d59bc6, entries=150, sequenceid=1097, filesize=11.9 K 2023-03-20 10:57:32,658 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:32,658 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:32,658 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:32,658 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:32,658 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:32,658 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:32,658 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:32,659 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/B/cf227877aeac411bbfd455262e019b35 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/cf227877aeac411bbfd455262e019b35 2023-03-20 10:57:32,659 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:32,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:32,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:32,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:32,660 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:32,660 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:32,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:32,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:32,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:32,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:32,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:32,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:32,661 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:32,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:32,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:32,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,662 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:32,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:32,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,663 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,664 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:32,664 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:32,664 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:32,664 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,664 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:32,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:32,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:32,664 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:32,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:32,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:32,664 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:32,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:32,664 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:32,664 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:32,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:32,664 DEBUG [Thread-1564] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:32,664 DEBUG [Thread-1564] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:32,668 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/cf227877aeac411bbfd455262e019b35, entries=150, sequenceid=1097, filesize=11.9 K 2023-03-20 10:57:32,669 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/C/9fd8492d5bdd4efb8d9d5eed33aeac40 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/9fd8492d5bdd4efb8d9d5eed33aeac40 2023-03-20 10:57:32,674 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/9fd8492d5bdd4efb8d9d5eed33aeac40, entries=150, sequenceid=1097, filesize=11.9 K 2023-03-20 10:57:32,675 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 72402f7d20e803d356627e208deb6fe2 in 138ms, sequenceid=1097, compaction requested=true 2023-03-20 10:57:32,675 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 72402f7d20e803d356627e208deb6fe2: 2023-03-20 10:57:32,676 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 10:57:32,676 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:57:32,676 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 10:57:32,676 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:57:32,676 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:57:32,691 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49256 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 10:57:32,691 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): 72402f7d20e803d356627e208deb6fe2/B is initiating minor compaction (all files) 2023-03-20 10:57:32,691 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 72402f7d20e803d356627e208deb6fe2/B in TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:32,691 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/a379fa2d36f44455be9902dc6e1d3e46, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/b0145fca1319491bbf81eb70b6dddfde, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/0ca593230ae54512968e6a7b5bf2c0b9, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/cf227877aeac411bbfd455262e019b35] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp, totalSize=48.1 K 2023-03-20 10:57:32,691 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51696 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 10:57:32,691 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting a379fa2d36f44455be9902dc6e1d3e46, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=1046, earliestPutTs=1679309850293 2023-03-20 10:57:32,691 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1912): 72402f7d20e803d356627e208deb6fe2/A is initiating minor compaction (all files) 2023-03-20 10:57:32,691 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 72402f7d20e803d356627e208deb6fe2/A in TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:32,692 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/bcc1226eb9f1486b9a920fd091ec88c6, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/b146f4acfc4f4bcd928a9ed18b614e17, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/f9456bb1ce61475f84976af7da9a70f1, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/24c544bc9cf942a08b5f9c6fe7d59bc6] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp, totalSize=50.5 K 2023-03-20 10:57:32,692 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting b0145fca1319491bbf81eb70b6dddfde, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1060, earliestPutTs=1679309850634 2023-03-20 10:57:32,692 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting bcc1226eb9f1486b9a920fd091ec88c6, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=1046, earliestPutTs=1679309850293 2023-03-20 10:57:32,692 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 0ca593230ae54512968e6a7b5bf2c0b9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1082, earliestPutTs=1679309851283 2023-03-20 10:57:32,692 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting b146f4acfc4f4bcd928a9ed18b614e17, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=1060, earliestPutTs=1679309850634 2023-03-20 10:57:32,693 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting cf227877aeac411bbfd455262e019b35, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1097, earliestPutTs=1679309852429 2023-03-20 10:57:32,694 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting f9456bb1ce61475f84976af7da9a70f1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1082, earliestPutTs=1679309851283 2023-03-20 10:57:32,695 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 24c544bc9cf942a08b5f9c6fe7d59bc6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1097, earliestPutTs=1679309852429 2023-03-20 10:57:32,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 72402f7d20e803d356627e208deb6fe2 2023-03-20 10:57:32,696 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 72402f7d20e803d356627e208deb6fe2 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-20 10:57:32,746 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=A 2023-03-20 10:57:32,746 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:57:32,746 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=B 2023-03-20 10:57:32,746 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:57:32,746 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=C 2023-03-20 10:57:32,746 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:57:32,765 DEBUG [Thread-1564] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:32,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:32,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:32,768 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): 72402f7d20e803d356627e208deb6fe2#B#compaction#250 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:57:32,768 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] throttle.PressureAwareThroughputController(145): 72402f7d20e803d356627e208deb6fe2#A#compaction#251 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:57:32,770 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:32,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 334 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55168 deadline: 1679309912759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,773 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:32,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55182 deadline: 1679309912769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,773 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:32,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 289 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55136 deadline: 1679309912770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,773 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:32,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 296 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55132 deadline: 1679309912770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:32,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:32,791 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:32,791 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:32,792 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:32,792 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:32,798 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:32,798 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:32,800 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,801 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,801 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:32,802 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:32,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:32,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:32,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:32,818 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:32,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:32,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:32,819 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:32,834 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:32,834 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:32,834 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:32,834 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:32,839 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/B/ecba251bf7194fa0bc095368ddf3c6fd as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/ecba251bf7194fa0bc095368ddf3c6fd 2023-03-20 10:57:32,842 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:32,842 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:32,842 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:32,842 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. started... 2023-03-20 10:57:32,842 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. as already flushing 2023-03-20 10:57:32,842 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:32,842 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:32,842 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:32,842 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,844 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 72402f7d20e803d356627e208deb6fe2/B of 72402f7d20e803d356627e208deb6fe2 into ecba251bf7194fa0bc095368ddf3c6fd(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:57:32,844 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 72402f7d20e803d356627e208deb6fe2: 2023-03-20 10:57:32,844 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2., storeName=72402f7d20e803d356627e208deb6fe2/B, priority=12, startTime=1679309852676; duration=0sec 2023-03-20 10:57:32,844 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:57:32,844 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 10:57:32,846 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:32,846 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:32,847 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49256 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 10:57:32,847 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): 72402f7d20e803d356627e208deb6fe2/C is initiating minor compaction (all files) 2023-03-20 10:57:32,847 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,847 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 72402f7d20e803d356627e208deb6fe2/C in TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:32,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:32,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:32,847 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:32,848 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/dc5782915282432d9cc78dda1b2a9b45, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/1d3848cec2e541259d2816cf2fd28d80, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/8d010a3e3b4a4a43bba443e353262666, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/9fd8492d5bdd4efb8d9d5eed33aeac40] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp, totalSize=48.1 K 2023-03-20 10:57:32,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:32,848 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/A/db92ab837db84ff4a34f63c1483ad329 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/db92ab837db84ff4a34f63c1483ad329 2023-03-20 10:57:32,848 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:32,848 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting dc5782915282432d9cc78dda1b2a9b45, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=1046, earliestPutTs=1679309850293 2023-03-20 10:57:32,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:32,848 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:32,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:32,849 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 1d3848cec2e541259d2816cf2fd28d80, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1060, earliestPutTs=1679309850634 2023-03-20 10:57:32,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,849 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 8d010a3e3b4a4a43bba443e353262666, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1082, earliestPutTs=1679309851283 2023-03-20 10:57:32,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:32,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:32,849 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 9fd8492d5bdd4efb8d9d5eed33aeac40, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1097, earliestPutTs=1679309852429 2023-03-20 10:57:32,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a9b23de[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:32,849 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:32,850 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:32,851 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:32,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:32,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:32,851 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:32,851 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,851 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:32,851 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:32,851 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:32,853 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:32,853 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,853 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:32,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:32,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:32,853 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:32,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:32,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:32,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:32,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:32,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:32,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:32,856 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:32,856 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:32,856 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:32,856 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:32,856 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:32,856 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:32,858 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:32,858 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:32,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:32,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:32,858 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:32,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:32,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:32,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:32,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:32,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:32,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:32,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:32,859 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:32,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:32,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:32,861 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,862 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): 72402f7d20e803d356627e208deb6fe2#C#compaction#252 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:57:32,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:32,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:32,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,868 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 72402f7d20e803d356627e208deb6fe2/A of 72402f7d20e803d356627e208deb6fe2 into db92ab837db84ff4a34f63c1483ad329(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:57:32,868 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 72402f7d20e803d356627e208deb6fe2: 2023-03-20 10:57:32,868 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2., storeName=72402f7d20e803d356627e208deb6fe2/A, priority=12, startTime=1679309852675; duration=0sec 2023-03-20 10:57:32,868 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:57:32,870 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,870 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:32,870 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:32,871 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:32,870 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:32,870 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:57:32,871 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:32,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:32,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:32,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:32,871 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:32,871 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:32,871 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:32,871 DEBUG [Thread-1564] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:32,871 DEBUG [Thread-1564] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:32,871 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:32,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:32,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:32,874 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:32,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 336 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55168 deadline: 1679309912872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,877 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:32,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 298 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55132 deadline: 1679309912874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,881 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:32,881 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:32,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55182 deadline: 1679309912875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 291 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55136 deadline: 1679309912875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,971 DEBUG [Thread-1564] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:32,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:32,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:32,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:32,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:32,975 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:32,975 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:32,975 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:32,975 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:32,975 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:32,975 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:32,976 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:32,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:32,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:32,976 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:32,977 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,977 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:32,977 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:32,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:32,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:32,977 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:32,978 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:32,978 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:32,978 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:32,978 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:32,978 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:32,988 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:32,988 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:32,992 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:32,993 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. started... 2023-03-20 10:57:32,993 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. as already flushing 2023-03-20 10:57:32,993 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:32,993 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:32,993 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:32,993 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,993 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:32,993 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:32,995 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,995 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:32,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:32,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:32,996 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:32,996 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:32,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:32,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:32,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:32,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:32,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:32,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:32,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d7668a2[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:32,998 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:32,998 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:32,999 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:32,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:32,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:32,999 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:32,999 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:32,999 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:33,000 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,000 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:33,002 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,002 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:33,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:33,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:33,002 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:33,002 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:33,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:33,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:33,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:33,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:33,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:33,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:33,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:33,006 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:33,006 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:33,006 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:33,006 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:33,006 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:33,012 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:33,012 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:33,012 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:33,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:33,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:33,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:33,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:33,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:33,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:33,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:33,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:33,014 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:33,015 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:33,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:33,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:33,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,018 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,018 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:33,018 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:33,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:33,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:33,018 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:33,018 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,018 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:33,018 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:33,018 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:33,018 DEBUG [Thread-1564] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:33,018 DEBUG [Thread-1564] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:33,018 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:33,018 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:33,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:33,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:33,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:33,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:33,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:33,020 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:57:33,079 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:33,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 338 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55168 deadline: 1679309913077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,080 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:33,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55132 deadline: 1679309913079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,087 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:33,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 293 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55136 deadline: 1679309913086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,087 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:33,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 303 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55182 deadline: 1679309913086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,118 DEBUG [Thread-1564] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:33,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:33,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:33,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:33,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:33,138 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:33,138 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:33,138 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:33,138 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:33,139 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:33,139 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:33,140 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:33,140 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:33,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:33,140 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,140 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:33,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:33,141 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:33,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:33,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:33,141 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:33,159 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:33,159 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:33,159 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:33,159 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:33,159 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:33,169 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:33,169 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:33,174 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:33,174 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. started... 2023-03-20 10:57:33,174 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. as already flushing 2023-03-20 10:57:33,174 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:33,175 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:33,175 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:33,175 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,175 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:33,175 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:33,176 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:33,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:33,176 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:33,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:33,177 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:33,177 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:33,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:33,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:33,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:33,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:33,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@fc5330[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:33,180 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:33,180 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:33,181 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:33,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:33,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:33,181 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:33,181 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:33,181 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:33,182 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,182 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:33,182 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:33,182 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:33,183 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:33,183 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:33,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:33,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:33,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:33,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:33,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:33,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:33,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:33,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:33,187 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:33,187 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=1118 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/A/dfa4dc201306418fb87832a9dc8d16e0 2023-03-20 10:57:33,187 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:33,187 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:33,187 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:33,187 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:33,195 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:33,195 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:33,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:33,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:33,195 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:33,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:33,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:33,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:33,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:33,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:33,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:33,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:33,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:33,197 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:33,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:33,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,200 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:33,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:33,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,205 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:33,205 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:33,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:33,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:33,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:33,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:33,206 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,206 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:33,206 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:33,206 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,206 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:33,206 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:33,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:33,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:33,206 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:33,206 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:33,206 DEBUG [Thread-1564] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:33,207 DEBUG [Thread-1564] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:33,251 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=1118 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/B/cbe2753a2ab942ea978c5de5485651bc 2023-03-20 10:57:33,307 DEBUG [Thread-1564] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:33,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:33,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:33,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:33,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:33,315 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:33,315 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:33,315 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:33,315 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:33,315 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:33,315 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:33,316 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:33,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:33,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:33,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:33,317 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,317 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:33,317 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,317 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:33,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:33,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:33,318 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:33,318 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:33,318 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:33,318 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:33,318 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:33,318 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:33,326 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:33,326 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:33,326 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:33,326 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. started... 2023-03-20 10:57:33,326 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. as already flushing 2023-03-20 10:57:33,326 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:33,326 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:33,326 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:33,326 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,326 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:33,326 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:33,328 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:33,329 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:33,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:33,329 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:33,329 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:33,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:33,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:33,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:33,330 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/C/da0c3fc6857243809f06c1bbf536d0b0 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/da0c3fc6857243809f06c1bbf536d0b0 2023-03-20 10:57:33,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:33,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:33,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@716a1fe7[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:33,331 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:33,331 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:33,332 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:33,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:33,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:33,332 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:33,332 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:33,332 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:33,332 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,332 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:33,333 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:33,333 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:33,333 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:33,333 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:33,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:33,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:33,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:33,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:33,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:33,334 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 72402f7d20e803d356627e208deb6fe2/C of 72402f7d20e803d356627e208deb6fe2 into da0c3fc6857243809f06c1bbf536d0b0(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:57:33,335 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 72402f7d20e803d356627e208deb6fe2: 2023-03-20 10:57:33,335 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2., storeName=72402f7d20e803d356627e208deb6fe2/C, priority=12, startTime=1679309852676; duration=0sec 2023-03-20 10:57:33,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:33,335 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:57:33,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:33,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:33,335 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:33,335 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:33,335 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:33,335 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:33,335 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:33,336 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:33,336 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:33,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:33,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:33,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:33,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:33,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:33,337 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:33,337 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:33,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:33,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:33,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:33,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:33,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:33,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:33,338 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:33,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:33,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,339 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:33,339 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:33,339 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:33,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:33,339 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:33,339 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:57:33,339 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:33,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:33,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:33,339 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:33,339 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,340 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:33,340 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:33,340 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:33,340 DEBUG [Thread-1564] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:33,340 DEBUG [Thread-1564] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:33,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:33,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:33,386 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:33,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 340 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55168 deadline: 1679309913382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,386 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:33,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55132 deadline: 1679309913382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,388 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:33,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 295 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55136 deadline: 1679309913388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,391 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:33,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 305 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55182 deadline: 1679309913389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,440 DEBUG [Thread-1564] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:33,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:33,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:33,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:33,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:33,443 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:33,443 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:33,443 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:33,443 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:33,443 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:33,444 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:33,444 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,444 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:33,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:33,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:33,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:33,445 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:33,445 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,445 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:33,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:33,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:33,445 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:33,445 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:33,445 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:33,446 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:33,446 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:33,446 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:33,446 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:33,446 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:33,446 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:33,446 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. started... 2023-03-20 10:57:33,446 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. as already flushing 2023-03-20 10:57:33,446 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:33,446 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:33,446 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:33,446 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,446 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:33,446 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:33,447 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,447 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:33,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:33,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:33,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:33,448 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:33,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:33,448 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:33,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:33,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:33,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:33,449 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:33,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@524c57c7[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:33,449 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:33,449 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:33,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:33,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:33,449 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:33,449 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:33,449 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:33,450 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,450 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:33,450 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,450 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:33,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:33,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:33,450 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:33,451 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:33,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:33,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:33,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:33,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:33,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:33,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:33,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:33,453 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:33,453 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:33,453 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:33,453 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:33,453 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:33,454 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:33,454 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:33,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:33,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:33,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:33,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:33,454 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:33,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:33,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:33,454 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:33,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:33,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:33,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:33,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:33,456 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:33,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:33,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:33,458 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,458 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:33,458 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:33,458 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,458 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:33,458 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:33,458 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:33,458 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:33,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:33,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:33,459 DEBUG [Thread-1564] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:33,459 DEBUG [Thread-1564] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:33,459 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:57:33,462 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:33,462 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:33,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:33,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:33,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:33,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:33,559 DEBUG [Thread-1564] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:33,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:33,563 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:33,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:33,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:33,564 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:33,564 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:33,565 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:33,565 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:33,565 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:33,565 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:33,566 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,566 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:33,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:33,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:33,566 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,566 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:33,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:33,567 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:33,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:33,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:33,567 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:33,570 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:33,570 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:33,570 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:33,570 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:33,570 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:33,570 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:33,570 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:33,570 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:33,571 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. started... 2023-03-20 10:57:33,571 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. as already flushing 2023-03-20 10:57:33,571 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:33,571 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:33,571 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:33,571 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,571 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:33,571 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:33,578 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:33,579 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:33,579 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:33,581 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:33,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:33,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:33,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:33,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:33,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:33,583 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:33,583 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1cb11654[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:33,583 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:33,583 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:33,583 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:33,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:33,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:33,584 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:33,584 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:33,584 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:33,585 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:33,585 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:33,585 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:33,586 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,586 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:33,586 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:33,586 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:33,586 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:33,586 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:33,586 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:33,586 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:33,588 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:33,588 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:33,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:33,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:33,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:33,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:33,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:33,588 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:33,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:33,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:33,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:33,589 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:33,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:33,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:33,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:33,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,590 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:33,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:33,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,592 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,592 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:33,593 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:33,593 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:33,593 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:33,593 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:33,593 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:33,593 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:33,593 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,593 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:33,593 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:33,593 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:33,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:33,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:33,593 DEBUG [Thread-1564] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:33,593 DEBUG [Thread-1564] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:33,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:33,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:33,673 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=1118 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/C/e63905300ba24d3db5312a6f7d9aa25c 2023-03-20 10:57:33,680 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/A/dfa4dc201306418fb87832a9dc8d16e0 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/dfa4dc201306418fb87832a9dc8d16e0 2023-03-20 10:57:33,685 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/dfa4dc201306418fb87832a9dc8d16e0, entries=250, sequenceid=1118, filesize=16.7 K 2023-03-20 10:57:33,686 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/B/cbe2753a2ab942ea978c5de5485651bc as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/cbe2753a2ab942ea978c5de5485651bc 2023-03-20 10:57:33,692 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/cbe2753a2ab942ea978c5de5485651bc, entries=150, sequenceid=1118, filesize=11.9 K 2023-03-20 10:57:33,693 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/C/e63905300ba24d3db5312a6f7d9aa25c as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/e63905300ba24d3db5312a6f7d9aa25c 2023-03-20 10:57:33,693 DEBUG [Thread-1564] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:33,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:33,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:33,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:33,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:33,698 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:33,699 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:33,699 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:33,699 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:33,699 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/e63905300ba24d3db5312a6f7d9aa25c, entries=150, sequenceid=1118, filesize=11.9 K 2023-03-20 10:57:33,699 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:33,699 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:33,700 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,700 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:33,700 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for 72402f7d20e803d356627e208deb6fe2 in 1004ms, sequenceid=1118, compaction requested=false 2023-03-20 10:57:33,700 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 72402f7d20e803d356627e208deb6fe2: 2023-03-20 10:57:33,700 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,700 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:33,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:33,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:33,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:33,701 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:33,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:33,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:33,702 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:33,702 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:33,702 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:33,702 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:33,702 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:33,702 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:33,702 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:33,702 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:33,702 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:33,703 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. started... 2023-03-20 10:57:33,703 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 72402f7d20e803d356627e208deb6fe2 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-20 10:57:33,703 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=A 2023-03-20 10:57:33,703 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:33,703 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=B 2023-03-20 10:57:33,703 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:33,703 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=C 2023-03-20 10:57:33,703 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:33,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 72402f7d20e803d356627e208deb6fe2 2023-03-20 10:57:33,892 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. as already flushing 2023-03-20 10:57:33,923 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:33,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 310 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55182 deadline: 1679309913919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,924 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679309853911, max timestamp=1679309853921Num uniques 859; 2023-03-20 10:57:33,924 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-20 10:57:33,924 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679309853911, max timestamp=1679309853921Num uniques 859; 2023-03-20 10:57:33,924 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-20 10:57:33,941 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:33,941 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:33,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 351 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55168 deadline: 1679309913937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 305 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55136 deadline: 1679309913938, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:33,942 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:33,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 312 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55132 deadline: 1679309913937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:34,002 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:34,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55118 deadline: 1679309913998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:34,003 DEBUG [Thread-1554] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8224 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2., hostname=jenkins-hbase20.apache.org,36041,1679309777062, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-20 10:57:34,026 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:34,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 312 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55182 deadline: 1679309914025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:34,043 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:34,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 307 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55136 deadline: 1679309914042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:34,044 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:34,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 353 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55168 deadline: 1679309914042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:34,045 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:34,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 314 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55132 deadline: 1679309914043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:34,122 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1132 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/A/094f7551839f4b0d908394149b12a713 2023-03-20 10:57:34,164 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1132 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/B/627bec0973d14a9d86b0d47559ab6973 2023-03-20 10:57:34,233 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:34,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 314 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55182 deadline: 1679309914229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:34,247 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:34,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 355 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55168 deadline: 1679309914245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:34,247 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:34,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 309 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55136 deadline: 1679309914247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:34,248 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:34,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 316 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55132 deadline: 1679309914247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:34,540 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:34,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 316 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55182 deadline: 1679309914535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:34,552 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:34,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 357 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55168 deadline: 1679309914549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:34,553 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:34,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 318 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55132 deadline: 1679309914550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:34,553 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:34,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 311 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55136 deadline: 1679309914551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:34,593 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1132 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/C/a91124e5d5c74ca9b57d3d4308daf4bd 2023-03-20 10:57:34,607 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/A/094f7551839f4b0d908394149b12a713 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/094f7551839f4b0d908394149b12a713 2023-03-20 10:57:34,614 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/094f7551839f4b0d908394149b12a713, entries=100, sequenceid=1132, filesize=9.5 K 2023-03-20 10:57:34,616 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/B/627bec0973d14a9d86b0d47559ab6973 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/627bec0973d14a9d86b0d47559ab6973 2023-03-20 10:57:34,625 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/627bec0973d14a9d86b0d47559ab6973, entries=100, sequenceid=1132, filesize=9.5 K 2023-03-20 10:57:34,626 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/C/a91124e5d5c74ca9b57d3d4308daf4bd as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/a91124e5d5c74ca9b57d3d4308daf4bd 2023-03-20 10:57:34,656 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/a91124e5d5c74ca9b57d3d4308daf4bd, entries=100, sequenceid=1132, filesize=9.5 K 2023-03-20 10:57:34,657 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 72402f7d20e803d356627e208deb6fe2 in 954ms, sequenceid=1132, compaction requested=true 2023-03-20 10:57:34,657 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 72402f7d20e803d356627e208deb6fe2: 2023-03-20 10:57:34,657 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:34,657 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:34,657 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:34,657 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:34,657 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:34,657 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:34,658 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:34,658 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:34,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:34,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:34,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:34,659 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:34,659 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:34,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:34,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:34,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:34,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:34,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:34,660 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:34,660 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e36b321[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:34,660 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:34,660 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:34,661 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:34,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:34,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:34,661 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:34,661 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:34,661 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:34,661 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:34,661 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:34,662 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:34,662 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:34,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:34,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:34,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:34,662 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:34,663 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:34,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:34,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:34,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:34,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:34,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:34,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:34,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:34,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:34,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:34,665 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:34,665 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:34,665 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:34,665 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:34,665 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:34,666 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:34,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:34,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:34,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:34,667 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:34,667 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:34,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:34,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:34,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:34,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:34,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:34,667 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:34,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:34,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:34,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:34,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:34,669 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:34,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:34,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:34,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:34,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:34,671 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:34,671 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:34,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:34,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:34,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:34,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:34,672 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:34,672 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:34,672 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:34,672 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:34,672 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:57:34,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:34,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:34,672 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:34,672 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:34,672 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:34,672 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:34,672 DEBUG [Thread-1564] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:34,672 DEBUG [Thread-1564] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:34,772 DEBUG [Thread-1564] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:34,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:34,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:34,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:34,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:34,786 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:34,786 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:34,786 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:34,786 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:34,786 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:34,786 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:34,787 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:34,787 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:34,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:34,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:34,787 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:34,787 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:34,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:34,788 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:34,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:34,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:34,788 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:34,794 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:34,794 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:34,794 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:34,794 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:34,794 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:34,802 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:34,802 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:34,802 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:34,802 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. started... 2023-03-20 10:57:34,804 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 72402f7d20e803d356627e208deb6fe2 3/3 column families, dataSize=134.18 KB heapSize=383.70 KB 2023-03-20 10:57:34,805 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=A 2023-03-20 10:57:34,805 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:57:34,805 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=B 2023-03-20 10:57:34,805 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:57:34,805 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=C 2023-03-20 10:57:34,805 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:57:34,862 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=1163 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/A/5db7f6101fa245ef8a962090dfd2c69b 2023-03-20 10:57:34,921 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=1163 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/B/cfb92fb4773c46d5822ee1dd2471037a 2023-03-20 10:57:34,968 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=62.62 KB at sequenceid=1163 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/C/18ec1c8a2bef437bac8fcb310f1210d7 2023-03-20 10:57:34,985 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/A/5db7f6101fa245ef8a962090dfd2c69b as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/5db7f6101fa245ef8a962090dfd2c69b 2023-03-20 10:57:34,989 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/5db7f6101fa245ef8a962090dfd2c69b, entries=150, sequenceid=1163, filesize=11.9 K 2023-03-20 10:57:34,994 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/B/cfb92fb4773c46d5822ee1dd2471037a as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/cfb92fb4773c46d5822ee1dd2471037a 2023-03-20 10:57:35,008 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/cfb92fb4773c46d5822ee1dd2471037a, entries=150, sequenceid=1163, filesize=11.9 K 2023-03-20 10:57:35,011 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/C/18ec1c8a2bef437bac8fcb310f1210d7 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/18ec1c8a2bef437bac8fcb310f1210d7 2023-03-20 10:57:35,028 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/18ec1c8a2bef437bac8fcb310f1210d7, entries=150, sequenceid=1163, filesize=11.9 K 2023-03-20 10:57:35,029 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~383.66 KB/392864, currentSize=0 B/0 for 72402f7d20e803d356627e208deb6fe2 in 227ms, sequenceid=1163, compaction requested=true 2023-03-20 10:57:35,029 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 72402f7d20e803d356627e208deb6fe2: 2023-03-20 10:57:35,029 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:35,029 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:35,030 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:35,030 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,030 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:35,030 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:35,031 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:35,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:35,032 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:35,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:35,032 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:35,032 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:35,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:35,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:35,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:35,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:35,035 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f020e27[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:35,035 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:35,035 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:35,036 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:35,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:35,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:35,036 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:35,036 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:35,036 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:35,037 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,037 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:35,038 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:35,038 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:35,038 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:35,039 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:35,039 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:35,039 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:35,039 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:35,039 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:35,039 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:35,040 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:35,040 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:35,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:35,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:35,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:35,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:35,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:35,041 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:35,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:35,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:35,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:35,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:35,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:35,043 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:35,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:35,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,044 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:35,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:35,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,048 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:35,048 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:35,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:35,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:35,049 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,049 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:35,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:35,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:35,049 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:35,049 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:35,049 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:35,049 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:35,049 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,049 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:35,049 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:35,049 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:35,049 DEBUG [Thread-1564] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:35,049 DEBUG [Thread-1564] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:35,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 72402f7d20e803d356627e208deb6fe2 2023-03-20 10:57:35,063 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 72402f7d20e803d356627e208deb6fe2 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-20 10:57:35,065 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=A 2023-03-20 10:57:35,065 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:35,065 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=B 2023-03-20 10:57:35,066 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:35,066 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=C 2023-03-20 10:57:35,066 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:35,125 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1174 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/A/c2a3e332bff4422b940636c87dc92247 2023-03-20 10:57:35,150 DEBUG [Thread-1564] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:35,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:35,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:35,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:35,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:35,154 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:35,154 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:35,154 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:35,154 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:35,155 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:35,155 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:35,155 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:35,155 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:35,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:35,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:35,156 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,156 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:35,156 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:35,157 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:35,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 319 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55136 deadline: 1679309915116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,158 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679309855108, max timestamp=1679309855117Num uniques 859; 2023-03-20 10:57:35,158 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-20 10:57:35,159 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679309855108, max timestamp=1679309855128Num uniques 859; 2023-03-20 10:57:35,159 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-03-20 10:57:35,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:35,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:35,160 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:35,160 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:35,160 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:35,160 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:35,160 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:35,160 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:35,160 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679309855108, max timestamp=1679309855128Num uniques 859; 2023-03-20 10:57:35,160 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-03-20 10:57:35,162 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:35,162 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:35,163 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:35,163 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. started... 2023-03-20 10:57:35,163 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. as already flushing 2023-03-20 10:57:35,163 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:35,163 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:35,163 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:35,163 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,163 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:35,163 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:35,170 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:35,171 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:35,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:35,171 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:35,171 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:35,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:35,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:35,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:35,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:35,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:35,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fc0886f[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:35,173 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:35,173 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:35,174 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:35,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:35,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:35,174 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:35,174 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:35,174 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:35,175 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,175 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:35,175 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:35,175 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:35,176 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:35,175 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:35,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:35,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:35,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:35,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:35,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:35,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:35,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:35,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:35,178 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:35,179 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:35,179 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:35,179 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:35,179 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:35,188 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:35,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:35,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:35,188 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:35,188 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:35,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:35,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:35,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:35,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:35,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:35,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:35,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:35,192 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:35,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:35,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:35,198 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:35,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:35,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,200 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,200 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:35,200 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:35,200 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:35,200 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:35,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:35,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:35,200 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,201 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:35,201 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:57:35,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:35,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:35,200 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:35,201 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:35,201 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:35,201 DEBUG [Thread-1564] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:35,201 DEBUG [Thread-1564] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:35,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:35,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:35,206 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:35,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 330 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55132 deadline: 1679309915198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,207 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:35,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 369 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55168 deadline: 1679309915200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,207 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:35,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 333 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55182 deadline: 1679309915202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,267 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:35,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 321 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55136 deadline: 1679309915266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,301 DEBUG [Thread-1564] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:35,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:35,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:35,309 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:35,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 371 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55168 deadline: 1679309915309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,314 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:35,314 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:35,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 332 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55132 deadline: 1679309915310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 335 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55182 deadline: 1679309915309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:35,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:35,322 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:35,322 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:35,322 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:35,322 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:35,322 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:35,322 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:35,323 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:35,323 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:35,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:35,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:35,324 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,324 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:35,324 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:35,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:35,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:35,329 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:35,346 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:35,346 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:35,346 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:35,346 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:35,346 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:35,355 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:35,355 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:35,355 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:35,357 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. started... 2023-03-20 10:57:35,357 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. as already flushing 2023-03-20 10:57:35,357 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:35,357 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:35,357 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:35,357 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,357 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:35,357 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:35,358 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:35,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:35,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:35,358 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:35,359 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:35,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:35,359 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:35,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:35,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:35,361 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:35,361 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7bd72075[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:35,361 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:35,361 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:35,361 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:35,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:35,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:35,362 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:35,362 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:35,362 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:35,362 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,362 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:35,362 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:35,362 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:35,362 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,363 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:35,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:35,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:35,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:35,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:35,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:35,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:35,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:35,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:35,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:35,367 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:35,367 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:35,367 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:35,367 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:35,367 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:35,369 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:35,369 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:35,369 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:35,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:35,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:35,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:35,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:35,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:35,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:35,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:35,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:35,373 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:35,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:35,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:35,375 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:35,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:35,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:35,386 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:35,386 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:35,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:35,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:35,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:35,387 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:35,387 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:35,387 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:35,387 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,387 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:35,387 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:35,387 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:35,386 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:35,387 DEBUG [Thread-1564] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:35,387 DEBUG [Thread-1564] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:35,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:35,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:35,387 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:57:35,473 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:35,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 323 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55136 deadline: 1679309915468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,487 DEBUG [Thread-1564] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:35,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:35,490 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:35,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:35,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:35,491 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:35,491 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:35,491 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:35,491 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:35,492 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:35,492 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:35,493 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:35,493 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:35,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:35,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:35,494 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,494 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:35,494 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:35,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:35,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:35,494 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:35,498 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:35,498 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:35,498 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:35,498 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:35,498 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:35,512 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:35,513 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:35,513 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:35,513 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. started... 2023-03-20 10:57:35,513 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. as already flushing 2023-03-20 10:57:35,513 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:35,513 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:35,513 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:35,513 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,513 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:35,513 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:35,515 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:35,515 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:35,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:35,515 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:35,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 373 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55168 deadline: 1679309915511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,516 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:35,516 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:35,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:35,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:35,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:35,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,522 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:35,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 337 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55182 deadline: 1679309915518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,522 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:35,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 334 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55132 deadline: 1679309915518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:35,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:35,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30b32b95[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:35,523 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:35,523 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:35,529 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:35,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:35,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:35,529 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:35,529 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:35,529 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:35,530 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,530 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:35,530 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:35,530 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:35,530 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:35,530 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:35,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:35,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:35,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:35,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:35,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:35,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:35,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:35,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:35,535 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:35,535 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:35,535 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:35,535 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:35,535 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:35,544 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1174 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/B/0ca86140ea82412080f4f0ca5a59c862 2023-03-20 10:57:35,545 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:35,545 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:35,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:35,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:35,545 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:35,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:35,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:35,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:35,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:35,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:35,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:35,547 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:35,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:35,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:35,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:35,549 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:35,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:35,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,553 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:35,553 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:35,553 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:35,553 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:35,553 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:57:35,553 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:35,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:35,553 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:35,553 DEBUG [Thread-1564] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:35,554 DEBUG [Thread-1564] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:35,553 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:35,554 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:35,554 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,554 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:35,554 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:35,554 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:35,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:35,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:35,609 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1174 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/C/231b62a970164153a414d4b00a76d059 2023-03-20 10:57:35,619 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/A/c2a3e332bff4422b940636c87dc92247 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/c2a3e332bff4422b940636c87dc92247 2023-03-20 10:57:35,634 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/c2a3e332bff4422b940636c87dc92247, entries=150, sequenceid=1174, filesize=11.9 K 2023-03-20 10:57:35,635 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/B/0ca86140ea82412080f4f0ca5a59c862 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/0ca86140ea82412080f4f0ca5a59c862 2023-03-20 10:57:35,654 DEBUG [Thread-1564] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:35,655 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/0ca86140ea82412080f4f0ca5a59c862, entries=150, sequenceid=1174, filesize=11.9 K 2023-03-20 10:57:35,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:35,659 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/C/231b62a970164153a414d4b00a76d059 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/231b62a970164153a414d4b00a76d059 2023-03-20 10:57:35,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:35,670 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/231b62a970164153a414d4b00a76d059, entries=150, sequenceid=1174, filesize=11.9 K 2023-03-20 10:57:35,671 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=126.68 KB/129720 for 72402f7d20e803d356627e208deb6fe2 in 608ms, sequenceid=1174, compaction requested=true 2023-03-20 10:57:35,671 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 72402f7d20e803d356627e208deb6fe2: 2023-03-20 10:57:35,671 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:57:35,671 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-20 10:57:35,671 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-20 10:57:35,671 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:57:35,671 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:57:35,673 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59134 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-20 10:57:35,673 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): 72402f7d20e803d356627e208deb6fe2/B is initiating minor compaction (all files) 2023-03-20 10:57:35,673 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 72402f7d20e803d356627e208deb6fe2/B in TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:35,673 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/ecba251bf7194fa0bc095368ddf3c6fd, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/cbe2753a2ab942ea978c5de5485651bc, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/627bec0973d14a9d86b0d47559ab6973, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/cfb92fb4773c46d5822ee1dd2471037a, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/0ca86140ea82412080f4f0ca5a59c862] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp, totalSize=57.7 K 2023-03-20 10:57:35,673 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 64014 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-20 10:57:35,674 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1912): 72402f7d20e803d356627e208deb6fe2/A is initiating minor compaction (all files) 2023-03-20 10:57:35,674 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 72402f7d20e803d356627e208deb6fe2/A in TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:35,674 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/db92ab837db84ff4a34f63c1483ad329, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/dfa4dc201306418fb87832a9dc8d16e0, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/094f7551839f4b0d908394149b12a713, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/5db7f6101fa245ef8a962090dfd2c69b, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/c2a3e332bff4422b940636c87dc92247] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp, totalSize=62.5 K 2023-03-20 10:57:35,674 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting ecba251bf7194fa0bc095368ddf3c6fd, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=1097, earliestPutTs=1679309852429 2023-03-20 10:57:35,677 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting db92ab837db84ff4a34f63c1483ad329, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=1097, earliestPutTs=1679309852429 2023-03-20 10:57:35,677 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting cbe2753a2ab942ea978c5de5485651bc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1118, earliestPutTs=1679309852695 2023-03-20 10:57:35,677 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting dfa4dc201306418fb87832a9dc8d16e0, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=1118, earliestPutTs=1679309852585 2023-03-20 10:57:35,677 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 627bec0973d14a9d86b0d47559ab6973, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=1132, earliestPutTs=1679309852752 2023-03-20 10:57:35,677 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 094f7551839f4b0d908394149b12a713, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=1132, earliestPutTs=1679309852752 2023-03-20 10:57:35,678 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting cfb92fb4773c46d5822ee1dd2471037a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1163, earliestPutTs=1679309853936 2023-03-20 10:57:35,678 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 5db7f6101fa245ef8a962090dfd2c69b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1163, earliestPutTs=1679309853936 2023-03-20 10:57:35,678 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 0ca86140ea82412080f4f0ca5a59c862, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1174, earliestPutTs=1679309855056 2023-03-20 10:57:35,678 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting c2a3e332bff4422b940636c87dc92247, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1174, earliestPutTs=1679309855056 2023-03-20 10:57:35,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:35,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:35,686 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:35,686 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:35,686 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:35,686 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:35,687 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:35,687 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:35,689 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:35,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:35,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:35,689 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:35,690 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,690 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:35,690 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:35,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:35,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:35,693 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:35,710 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:35,710 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:35,710 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:35,710 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:35,710 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:35,722 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:35,722 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:35,729 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:35,729 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. started... 2023-03-20 10:57:35,730 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): 72402f7d20e803d356627e208deb6fe2#B#compaction#264 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:57:35,730 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 72402f7d20e803d356627e208deb6fe2 3/3 column families, dataSize=126.68 KB heapSize=363.23 KB 2023-03-20 10:57:35,730 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=A 2023-03-20 10:57:35,730 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:57:35,730 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=B 2023-03-20 10:57:35,730 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:57:35,730 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=C 2023-03-20 10:57:35,730 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:57:35,733 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] throttle.PressureAwareThroughputController(145): 72402f7d20e803d356627e208deb6fe2#A#compaction#265 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:57:35,784 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/A/71cfb0027f744c52a2e1d4f24e14b524 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/71cfb0027f744c52a2e1d4f24e14b524 2023-03-20 10:57:35,784 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. as already flushing 2023-03-20 10:57:35,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 72402f7d20e803d356627e208deb6fe2 2023-03-20 10:57:35,791 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 72402f7d20e803d356627e208deb6fe2/A of 72402f7d20e803d356627e208deb6fe2 into 71cfb0027f744c52a2e1d4f24e14b524(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:57:35,791 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 72402f7d20e803d356627e208deb6fe2: 2023-03-20 10:57:35,791 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2., storeName=72402f7d20e803d356627e208deb6fe2/A, priority=11, startTime=1679309855671; duration=0sec 2023-03-20 10:57:35,792 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:57:35,792 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-20 10:57:35,794 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59134 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-20 10:57:35,794 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1912): 72402f7d20e803d356627e208deb6fe2/C is initiating minor compaction (all files) 2023-03-20 10:57:35,794 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 72402f7d20e803d356627e208deb6fe2/C in TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:35,794 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/da0c3fc6857243809f06c1bbf536d0b0, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/e63905300ba24d3db5312a6f7d9aa25c, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/a91124e5d5c74ca9b57d3d4308daf4bd, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/18ec1c8a2bef437bac8fcb310f1210d7, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/231b62a970164153a414d4b00a76d059] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp, totalSize=57.7 K 2023-03-20 10:57:35,794 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting da0c3fc6857243809f06c1bbf536d0b0, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=1097, earliestPutTs=1679309852429 2023-03-20 10:57:35,796 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting e63905300ba24d3db5312a6f7d9aa25c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1118, earliestPutTs=1679309852695 2023-03-20 10:57:35,796 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting a91124e5d5c74ca9b57d3d4308daf4bd, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=1132, earliestPutTs=1679309852752 2023-03-20 10:57:35,796 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 18ec1c8a2bef437bac8fcb310f1210d7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1163, earliestPutTs=1679309853936 2023-03-20 10:57:35,796 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 231b62a970164153a414d4b00a76d059, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1174, earliestPutTs=1679309855056 2023-03-20 10:57:35,818 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=1210 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/A/e410618932b145a2803c2b40a7b1d850 2023-03-20 10:57:35,820 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] throttle.PressureAwareThroughputController(145): 72402f7d20e803d356627e208deb6fe2#C#compaction#267 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:57:35,829 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:35,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 336 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55132 deadline: 1679309915824, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,830 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:35,830 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:35,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 331 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55136 deadline: 1679309915824, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 339 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55182 deadline: 1679309915825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,833 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:35,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 377 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55168 deadline: 1679309915827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,835 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/C/956a1308fd7a4dadbc1c71ea93415005 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/956a1308fd7a4dadbc1c71ea93415005 2023-03-20 10:57:35,842 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 72402f7d20e803d356627e208deb6fe2/C of 72402f7d20e803d356627e208deb6fe2 into 956a1308fd7a4dadbc1c71ea93415005(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:57:35,842 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 72402f7d20e803d356627e208deb6fe2: 2023-03-20 10:57:35,842 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2., storeName=72402f7d20e803d356627e208deb6fe2/C, priority=11, startTime=1679309855671; duration=0sec 2023-03-20 10:57:35,842 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:57:35,935 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:35,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 333 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55136 deadline: 1679309915932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:35,941 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:35,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 379 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55168 deadline: 1679309915938, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:36,138 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:36,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 335 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55136 deadline: 1679309916138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:36,146 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:36,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 381 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55168 deadline: 1679309916143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:36,165 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/B/49b9e7aae31245e3b151c438355eed89 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/49b9e7aae31245e3b151c438355eed89 2023-03-20 10:57:36,171 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 72402f7d20e803d356627e208deb6fe2/B of 72402f7d20e803d356627e208deb6fe2 into 49b9e7aae31245e3b151c438355eed89(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:57:36,171 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 72402f7d20e803d356627e208deb6fe2: 2023-03-20 10:57:36,171 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2., storeName=72402f7d20e803d356627e208deb6fe2/B, priority=11, startTime=1679309855671; duration=0sec 2023-03-20 10:57:36,172 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:57:36,248 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.09 KB at sequenceid=1210 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/B/7f086cd43fb8446aaae928443472e838 2023-03-20 10:57:36,333 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:36,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 341 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55182 deadline: 1679309916331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:36,334 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:36,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 338 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55132 deadline: 1679309916333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:36,444 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:36,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 337 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55136 deadline: 1679309916442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:36,454 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:36,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 383 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55168 deadline: 1679309916452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:36,675 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.09 KB at sequenceid=1210 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/C/fc56513ac898486eb93451797490f7c6 2023-03-20 10:57:36,683 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/A/e410618932b145a2803c2b40a7b1d850 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/e410618932b145a2803c2b40a7b1d850 2023-03-20 10:57:36,689 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/e410618932b145a2803c2b40a7b1d850, entries=150, sequenceid=1210, filesize=11.9 K 2023-03-20 10:57:36,691 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/B/7f086cd43fb8446aaae928443472e838 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/7f086cd43fb8446aaae928443472e838 2023-03-20 10:57:36,700 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/7f086cd43fb8446aaae928443472e838, entries=150, sequenceid=1210, filesize=11.9 K 2023-03-20 10:57:36,702 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/C/fc56513ac898486eb93451797490f7c6 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/fc56513ac898486eb93451797490f7c6 2023-03-20 10:57:36,746 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/fc56513ac898486eb93451797490f7c6, entries=150, sequenceid=1210, filesize=11.9 K 2023-03-20 10:57:36,747 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~126.68 KB/129720, heapSize ~363.19 KB/371904, currentSize=53.67 KB/54960 for 72402f7d20e803d356627e208deb6fe2 in 1017ms, sequenceid=1210, compaction requested=false 2023-03-20 10:57:36,747 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 72402f7d20e803d356627e208deb6fe2: 2023-03-20 10:57:36,748 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:36,748 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:36,748 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:36,748 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:36,748 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:36,748 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:36,749 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:36,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:36,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:36,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:36,749 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:36,749 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:36,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:36,749 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:36,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:36,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:36,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:36,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:36,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:36,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f0f6892[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:36,755 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:36,755 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:36,762 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:36,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:36,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:36,762 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:36,763 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:36,763 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:36,763 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:36,763 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:36,770 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:36,770 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:36,770 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:36,770 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:36,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:36,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:36,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:36,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:36,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:36,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:36,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:36,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:36,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:36,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:36,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:36,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:36,783 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:36,783 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:36,783 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:36,783 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:36,783 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:36,784 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:36,784 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:36,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:36,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:36,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:36,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:36,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:36,784 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:36,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:36,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:36,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:36,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:36,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:36,785 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:36,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:36,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:36,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:36,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:36,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:36,787 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:36,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:36,795 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:36,795 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:36,795 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:36,795 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:36,795 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:36,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:36,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:36,795 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:36,795 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:36,795 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:36,796 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:36,796 DEBUG [Thread-1564] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:36,796 DEBUG [Thread-1564] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:36,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:36,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:36,795 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:36,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:36,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:36,898 DEBUG [Thread-1564] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:36,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:36,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:36,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:36,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:36,902 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:36,902 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:36,902 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:36,902 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:36,903 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:36,903 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:36,904 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:36,904 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:36,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:36,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:36,904 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:36,904 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:36,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:36,904 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:36,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:36,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:36,905 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:36,905 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:36,905 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:36,905 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:36,905 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:36,905 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:36,905 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:36,905 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:36,905 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. started... 2023-03-20 10:57:36,906 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 72402f7d20e803d356627e208deb6fe2 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-20 10:57:36,906 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=A 2023-03-20 10:57:36,906 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:36,906 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=B 2023-03-20 10:57:36,906 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:36,906 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=C 2023-03-20 10:57:36,906 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:36,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 72402f7d20e803d356627e208deb6fe2 2023-03-20 10:57:36,951 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. as already flushing 2023-03-20 10:57:37,001 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:37,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 393 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55168 deadline: 1679309916996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:37,001 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679309856988, max timestamp=1679309856998Num uniques 859; 2023-03-20 10:57:37,001 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-20 10:57:37,003 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679309856988, max timestamp=1679309856998Num uniques 859; 2023-03-20 10:57:37,003 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-20 10:57:37,003 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679309856988, max timestamp=1679309856998Num uniques 859; 2023-03-20 10:57:37,004 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-03-20 10:57:37,076 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:37,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 364 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55136 deadline: 1679309917074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:37,104 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:37,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 395 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55168 deadline: 1679309917102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:37,180 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:37,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 366 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55136 deadline: 1679309917177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:37,308 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:37,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 397 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55168 deadline: 1679309917306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:37,319 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1224 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/A/bad8240ce9894461bb7627f369a25a8a 2023-03-20 10:57:37,339 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:37,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 343 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55182 deadline: 1679309917337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:37,352 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1224 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/B/5b60071ef32b46abbd7f294650fa673f 2023-03-20 10:57:37,358 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:37,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 340 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55132 deadline: 1679309917354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:37,387 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:37,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 368 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55136 deadline: 1679309917384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:37,402 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1224 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/C/621afc1df2e94efb88c98f0f6814fe22 2023-03-20 10:57:37,407 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/A/bad8240ce9894461bb7627f369a25a8a as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/bad8240ce9894461bb7627f369a25a8a 2023-03-20 10:57:37,418 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/bad8240ce9894461bb7627f369a25a8a, entries=150, sequenceid=1224, filesize=11.9 K 2023-03-20 10:57:37,419 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/B/5b60071ef32b46abbd7f294650fa673f as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/5b60071ef32b46abbd7f294650fa673f 2023-03-20 10:57:37,426 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/5b60071ef32b46abbd7f294650fa673f, entries=150, sequenceid=1224, filesize=11.9 K 2023-03-20 10:57:37,431 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/C/621afc1df2e94efb88c98f0f6814fe22 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/621afc1df2e94efb88c98f0f6814fe22 2023-03-20 10:57:37,447 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/621afc1df2e94efb88c98f0f6814fe22, entries=150, sequenceid=1224, filesize=11.9 K 2023-03-20 10:57:37,448 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.08 KB/130125 for 72402f7d20e803d356627e208deb6fe2 in 543ms, sequenceid=1224, compaction requested=true 2023-03-20 10:57:37,448 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 72402f7d20e803d356627e208deb6fe2: 2023-03-20 10:57:37,448 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:37,448 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:37,448 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:37,448 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:37,450 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:37,450 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:37,451 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:37,451 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:37,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:37,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:37,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:37,451 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:37,451 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:37,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:37,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:37,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:37,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:37,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:37,453 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:37,453 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@146704ec[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:37,453 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:37,453 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:37,454 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:37,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:37,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:37,454 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:37,454 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:37,454 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:37,454 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:37,454 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:37,455 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:37,455 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:37,455 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:37,455 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:37,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:37,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:37,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:37,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:37,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:37,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:37,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:37,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:37,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:37,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:37,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:37,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:37,459 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:37,459 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:37,459 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:37,459 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:37,459 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:37,460 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:37,460 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:37,460 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:37,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:37,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:37,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:37,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:37,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:37,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:37,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:37,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:37,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:37,461 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:37,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:37,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:37,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:37,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:37,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:37,466 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:37,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:37,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:37,468 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:37,468 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:37,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:37,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:37,468 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:37,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:37,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:37,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:37,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:37,469 DEBUG [Thread-1564] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:37,469 DEBUG [Thread-1564] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:37,469 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:37,469 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:37,469 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:37,469 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:37,469 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:37,469 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:37,468 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:37,569 DEBUG [Thread-1564] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:37,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:37,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:37,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:37,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:37,575 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:37,575 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:37,575 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:37,575 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:37,576 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:37,576 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:37,577 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:37,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:37,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:37,577 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:37,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:37,577 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:37,577 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:37,577 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:37,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:37,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:37,578 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:37,582 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:37,582 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:37,582 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:37,582 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:37,582 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:37,586 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:37,586 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:37,586 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:37,586 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. started... 2023-03-20 10:57:37,586 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 72402f7d20e803d356627e208deb6fe2 3/3 column families, dataSize=127.08 KB heapSize=364.15 KB 2023-03-20 10:57:37,587 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=A 2023-03-20 10:57:37,587 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:57:37,587 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=B 2023-03-20 10:57:37,587 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:57:37,587 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=C 2023-03-20 10:57:37,587 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:57:37,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 72402f7d20e803d356627e208deb6fe2 2023-03-20 10:57:37,612 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. as already flushing 2023-03-20 10:57:37,650 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:37,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 407 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55168 deadline: 1679309917646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:37,692 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:37,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 370 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55136 deadline: 1679309917689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:37,754 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:37,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 409 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55168 deadline: 1679309917751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:37,959 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:37,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 411 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55168 deadline: 1679309917956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:38,016 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=1260 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/A/39d8474f08484aac92143096dcd5acf3 2023-03-20 10:57:38,197 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:38,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 372 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55136 deadline: 1679309918194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:38,263 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:38,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 413 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55168 deadline: 1679309918261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:38,437 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.09 KB at sequenceid=1260 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/B/efa0358c15bb4decae825b6625aa1c95 2023-03-20 10:57:38,768 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:38,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 415 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55168 deadline: 1679309918766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:38,848 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=1260 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/C/ed51c9e3b10442ce9955af7de79a3232 2023-03-20 10:57:38,861 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/A/39d8474f08484aac92143096dcd5acf3 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/39d8474f08484aac92143096dcd5acf3 2023-03-20 10:57:38,868 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/39d8474f08484aac92143096dcd5acf3, entries=150, sequenceid=1260, filesize=11.9 K 2023-03-20 10:57:38,869 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/B/efa0358c15bb4decae825b6625aa1c95 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/efa0358c15bb4decae825b6625aa1c95 2023-03-20 10:57:38,882 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/efa0358c15bb4decae825b6625aa1c95, entries=150, sequenceid=1260, filesize=11.9 K 2023-03-20 10:57:38,883 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/C/ed51c9e3b10442ce9955af7de79a3232 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/ed51c9e3b10442ce9955af7de79a3232 2023-03-20 10:57:38,889 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/ed51c9e3b10442ce9955af7de79a3232, entries=150, sequenceid=1260, filesize=11.9 K 2023-03-20 10:57:38,889 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.08 KB/130125, heapSize ~364.10 KB/372840, currentSize=53.67 KB/54960 for 72402f7d20e803d356627e208deb6fe2 in 1303ms, sequenceid=1260, compaction requested=true 2023-03-20 10:57:38,890 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 72402f7d20e803d356627e208deb6fe2: 2023-03-20 10:57:38,890 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:38,890 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:38,890 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:38,890 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:38,890 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:38,890 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:38,902 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:38,902 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:38,902 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:38,906 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:38,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:38,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:38,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:38,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:38,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:38,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:38,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:38,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:38,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:38,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56f15a9d[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:38,908 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:38,908 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:38,909 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:38,909 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:38,909 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:38,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:38,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:38,910 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:38,910 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:38,910 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:38,911 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:38,911 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:38,911 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:38,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:38,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:38,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:38,911 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:38,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:38,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:38,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:38,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:38,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:38,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:38,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:38,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:38,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:38,915 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:38,915 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:38,915 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:38,915 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:38,915 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:38,920 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:38,920 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:38,920 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:38,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:38,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:38,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:38,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:38,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:38,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:38,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:38,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:38,921 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:38,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:38,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:38,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:38,923 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:38,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:38,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:38,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:38,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:38,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:38,925 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:38,926 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:38,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:38,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:38,926 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:38,926 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:38,926 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:38,926 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:38,926 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:38,926 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:38,926 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:38,926 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:38,926 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:38,926 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:38,926 DEBUG [Thread-1564] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:38,926 DEBUG [Thread-1564] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:38,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:38,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:38,927 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:57:39,026 DEBUG [Thread-1564] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:39,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:39,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:39,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:39,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:39,033 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:39,033 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:39,033 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:39,033 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:39,033 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:39,033 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:39,034 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:39,034 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:39,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:39,035 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,035 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:39,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:39,035 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:39,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:39,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:39,036 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:39,042 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:39,042 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:39,042 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:39,042 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:39,042 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:39,050 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:39,050 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:39,050 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:39,050 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. started... 2023-03-20 10:57:39,050 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 72402f7d20e803d356627e208deb6fe2 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-20 10:57:39,050 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=A 2023-03-20 10:57:39,050 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:39,051 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=B 2023-03-20 10:57:39,051 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:39,051 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=C 2023-03-20 10:57:39,051 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:39,063 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1271 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/A/ef0713c23f0f4cf58f57214d4ae48989 2023-03-20 10:57:39,085 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1271 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/B/850deb7da4604dd2b983c0a1c5f7837f 2023-03-20 10:57:39,100 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1271 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/C/2420f8e532e843aabee79e586d364b59 2023-03-20 10:57:39,103 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/A/ef0713c23f0f4cf58f57214d4ae48989 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/ef0713c23f0f4cf58f57214d4ae48989 2023-03-20 10:57:39,107 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/ef0713c23f0f4cf58f57214d4ae48989, entries=150, sequenceid=1271, filesize=11.9 K 2023-03-20 10:57:39,107 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/B/850deb7da4604dd2b983c0a1c5f7837f as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/850deb7da4604dd2b983c0a1c5f7837f 2023-03-20 10:57:39,111 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/850deb7da4604dd2b983c0a1c5f7837f, entries=150, sequenceid=1271, filesize=11.9 K 2023-03-20 10:57:39,111 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/C/2420f8e532e843aabee79e586d364b59 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/2420f8e532e843aabee79e586d364b59 2023-03-20 10:57:39,115 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/2420f8e532e843aabee79e586d364b59, entries=150, sequenceid=1271, filesize=11.9 K 2023-03-20 10:57:39,116 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=0 B/0 for 72402f7d20e803d356627e208deb6fe2 in 66ms, sequenceid=1271, compaction requested=true 2023-03-20 10:57:39,116 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 72402f7d20e803d356627e208deb6fe2: 2023-03-20 10:57:39,116 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:39,116 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:39,116 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:39,116 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,116 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:39,116 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:39,117 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,117 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:39,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:39,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:39,117 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:39,118 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:39,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:39,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:39,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:39,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:39,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:39,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fb73c55[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:39,119 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:39,119 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:39,121 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:39,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:39,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:39,121 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:39,121 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:39,121 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:39,121 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,121 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:39,122 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:39,122 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:39,122 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:39,122 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:39,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:39,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:39,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:39,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:39,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:39,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:39,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:39,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:39,126 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:39,126 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:39,126 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:39,126 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:39,126 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:39,127 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:39,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:39,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:39,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:39,127 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:39,127 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:39,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:39,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:39,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:39,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:39,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:39,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:39,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:39,134 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:39,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:39,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:39,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:39,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,143 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,144 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:39,144 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:39,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:39,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:39,144 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,144 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:39,144 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:39,144 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,144 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:39,144 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:39,145 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:39,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:39,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:39,146 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:39,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:39,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:39,146 DEBUG [Thread-1564] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:39,146 DEBUG [Thread-1564] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:39,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 72402f7d20e803d356627e208deb6fe2 2023-03-20 10:57:39,222 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 72402f7d20e803d356627e208deb6fe2 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-20 10:57:39,223 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=A 2023-03-20 10:57:39,223 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:39,223 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=B 2023-03-20 10:57:39,223 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:39,223 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=C 2023-03-20 10:57:39,223 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:39,231 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=1281 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/A/16fcc391c6144109bafd8f18c94700c0 2023-03-20 10:57:39,246 DEBUG [Thread-1564] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:39,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:39,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:39,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:39,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:39,249 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:39,249 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:39,249 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:39,249 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:39,249 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:39,249 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:39,250 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:39,250 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:39,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:39,250 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,250 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:39,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:39,250 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:39,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:39,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:39,250 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:39,251 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:39,251 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:39,251 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:39,251 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:39,251 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:39,251 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:39,251 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:39,251 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:39,251 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. started... 2023-03-20 10:57:39,251 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. as already flushing 2023-03-20 10:57:39,251 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:39,251 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:39,251 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:39,251 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,251 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:39,251 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:39,252 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,252 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:39,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:39,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:39,252 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:39,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:39,252 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:39,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:39,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:39,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:39,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:39,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b7890c9[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:39,253 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:39,253 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:39,254 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:39,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:39,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:39,254 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:39,254 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:39,254 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:39,254 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,254 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:39,255 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,255 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:39,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:39,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:39,255 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:39,255 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:39,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:39,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:39,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:39,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:39,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:39,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:39,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:39,256 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:39,256 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:39,256 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:39,256 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:39,257 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:39,257 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:39,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:39,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:39,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:39,257 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:39,257 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:39,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:39,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:39,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:39,258 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:39,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:39,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:39,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:39,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:39,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:39,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,259 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:39,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:39,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,260 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,260 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:39,260 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:39,260 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:39,260 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:39,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:39,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:39,260 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:39,260 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,260 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:57:39,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:39,260 DEBUG [Thread-1564] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:39,260 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:39,260 DEBUG [Thread-1564] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:39,261 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:39,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:39,261 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:39,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:39,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:39,307 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679309859277, max timestamp=1679309859305Num uniques 859; 2023-03-20 10:57:39,307 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-20 10:57:39,307 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679309859277, max timestamp=1679309859305Num uniques 859; 2023-03-20 10:57:39,307 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-20 10:57:39,307 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679309859277, max timestamp=1679309859305Num uniques 859; 2023-03-20 10:57:39,307 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-20 10:57:39,361 DEBUG [Thread-1564] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:39,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:39,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:39,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:39,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:39,363 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:39,364 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:39,364 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:39,364 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:39,364 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:39,364 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:39,365 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:39,365 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:39,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:39,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:39,365 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,365 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:39,365 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:39,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:39,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:39,367 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:39,371 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:39,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 411 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55136 deadline: 1679309919368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,371 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:39,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 342 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55132 deadline: 1679309919368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,372 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:39,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 349 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55182 deadline: 1679309919370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,374 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:39,374 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:39,374 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:39,374 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:39,375 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:39,375 DEBUG [Thread-1558] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4174 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2., hostname=jenkins-hbase20.apache.org,36041,1679309777062, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-20 10:57:39,376 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:39,376 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:39,376 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:39,376 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. started... 2023-03-20 10:57:39,376 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. as already flushing 2023-03-20 10:57:39,376 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:39,376 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:39,376 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:39,376 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,376 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:39,376 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:39,378 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:39,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:39,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:39,379 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:39,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:39,379 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:39,379 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:39,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:39,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:39,380 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:39,380 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5714bb2e[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:39,380 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:39,380 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:39,381 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:39,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:39,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:39,381 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:39,381 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:39,381 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:39,381 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,381 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:39,383 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:39,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:39,383 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:39,383 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:39,383 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:39,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:39,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:39,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:39,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:39,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:39,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:39,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:39,385 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:39,385 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:39,385 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:39,385 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:39,385 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:39,387 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:39,387 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:39,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:39,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:39,387 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:39,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:39,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:39,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:39,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:39,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:39,387 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:39,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:39,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:39,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:39,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:39,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,388 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:39,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:39,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,390 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,390 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:39,390 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:39,390 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,390 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:39,390 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:39,390 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:39,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:39,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:39,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:39,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:39,390 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:39,390 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:39,391 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:39,391 DEBUG [Thread-1564] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:39,391 DEBUG [Thread-1564] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:39,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:39,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:39,474 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:39,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 413 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55136 deadline: 1679309919472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,475 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:39,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 351 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55182 deadline: 1679309919472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,491 DEBUG [Thread-1564] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:39,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:39,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:39,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:39,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:39,496 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:39,497 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:39,497 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:39,497 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:39,497 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:39,497 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:39,498 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,498 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:39,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:39,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:39,499 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,499 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:39,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:39,499 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:39,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:39,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:39,500 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:39,500 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:39,500 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:39,500 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:39,501 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:39,501 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:39,501 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:39,501 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:39,501 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:39,501 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. started... 2023-03-20 10:57:39,501 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. as already flushing 2023-03-20 10:57:39,501 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:39,501 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:39,501 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:39,501 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,501 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:39,501 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:39,503 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,503 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:39,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:39,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:39,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:39,503 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:39,503 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:39,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:39,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:39,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:39,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:39,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@bed26e3[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:39,504 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:39,504 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:39,505 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:39,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:39,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:39,505 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:39,505 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:39,505 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:39,508 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:39,508 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:39,508 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:39,509 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,510 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:39,510 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:39,510 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:39,510 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:39,510 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:39,510 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:39,510 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:39,512 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:39,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:39,512 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:39,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:39,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:39,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:39,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:39,513 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:39,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:39,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:39,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:39,514 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:39,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:39,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:39,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:39,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:39,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:39,517 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,528 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,529 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:39,529 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:39,529 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,529 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:39,530 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:39,530 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:39,530 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:39,530 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:39,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:39,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:39,533 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:39,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:39,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:39,533 DEBUG [Thread-1564] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:39,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:39,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:39,533 DEBUG [Thread-1564] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:39,634 DEBUG [Thread-1564] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:39,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:39,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:39,640 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=1281 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/B/9faa5da879f84c549e9fa3013c79be06 2023-03-20 10:57:39,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:39,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:39,650 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:39,650 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:39,650 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:39,650 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:39,651 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:39,651 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:39,651 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:39,651 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:39,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:39,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:39,652 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,652 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:39,652 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:39,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:39,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:39,653 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:39,654 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:39,654 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:39,654 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:39,654 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:39,654 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:39,658 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:39,658 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:39,658 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:39,658 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. started... 2023-03-20 10:57:39,658 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. as already flushing 2023-03-20 10:57:39,658 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:39,658 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:39,658 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:39,658 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,658 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:39,658 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:39,662 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:39,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:39,662 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:39,663 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:39,663 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:39,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:39,663 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=1281 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/C/b836f7736c9c4af6831ca8fedbea3b2c 2023-03-20 10:57:39,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:39,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:39,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:39,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:39,665 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:39,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4dfef65f[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:39,665 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:39,666 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:39,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:39,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:39,666 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:39,666 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:39,666 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:39,666 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,666 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:39,668 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:39,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:39,669 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:39,669 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:39,669 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:39,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:39,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:39,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:39,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:39,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:39,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:39,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:39,672 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:39,672 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:39,672 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:39,672 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:39,672 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:39,673 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:39,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:39,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:39,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:39,673 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:39,674 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:39,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:39,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:39,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:39,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:39,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:39,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:39,676 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/A/16fcc391c6144109bafd8f18c94700c0 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/16fcc391c6144109bafd8f18c94700c0 2023-03-20 10:57:39,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:39,677 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:39,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:39,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,679 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:39,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 415 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55136 deadline: 1679309919677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:39,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:39,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,683 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:39,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 353 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55182 deadline: 1679309919682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,684 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,686 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,686 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:39,686 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:39,686 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:39,686 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:39,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:39,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:39,686 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,686 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:39,686 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:39,686 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:39,686 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:39,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:39,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:39,687 DEBUG [Thread-1564] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:39,687 DEBUG [Thread-1564] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:39,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:39,687 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/16fcc391c6144109bafd8f18c94700c0, entries=200, sequenceid=1281, filesize=14.3 K 2023-03-20 10:57:39,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:39,690 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/B/9faa5da879f84c549e9fa3013c79be06 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/9faa5da879f84c549e9fa3013c79be06 2023-03-20 10:57:39,693 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/9faa5da879f84c549e9fa3013c79be06, entries=150, sequenceid=1281, filesize=11.9 K 2023-03-20 10:57:39,694 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/C/b836f7736c9c4af6831ca8fedbea3b2c as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/b836f7736c9c4af6831ca8fedbea3b2c 2023-03-20 10:57:39,697 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/b836f7736c9c4af6831ca8fedbea3b2c, entries=150, sequenceid=1281, filesize=11.9 K 2023-03-20 10:57:39,698 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 72402f7d20e803d356627e208deb6fe2 in 476ms, sequenceid=1281, compaction requested=true 2023-03-20 10:57:39,698 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 72402f7d20e803d356627e208deb6fe2: 2023-03-20 10:57:39,698 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:57:39,698 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-20 10:57:39,699 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-20 10:57:39,699 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:57:39,699 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:57:39,701 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 76374 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-20 10:57:39,701 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1912): 72402f7d20e803d356627e208deb6fe2/A is initiating minor compaction (all files) 2023-03-20 10:57:39,701 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 72402f7d20e803d356627e208deb6fe2/A in TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:39,701 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/71cfb0027f744c52a2e1d4f24e14b524, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/e410618932b145a2803c2b40a7b1d850, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/bad8240ce9894461bb7627f369a25a8a, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/39d8474f08484aac92143096dcd5acf3, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/ef0713c23f0f4cf58f57214d4ae48989, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/16fcc391c6144109bafd8f18c94700c0] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp, totalSize=74.6 K 2023-03-20 10:57:39,702 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73934 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-20 10:57:39,702 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 71cfb0027f744c52a2e1d4f24e14b524, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=1174, earliestPutTs=1679309855056 2023-03-20 10:57:39,702 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): 72402f7d20e803d356627e208deb6fe2/B is initiating minor compaction (all files) 2023-03-20 10:57:39,702 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 72402f7d20e803d356627e208deb6fe2/B in TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:39,702 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/49b9e7aae31245e3b151c438355eed89, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/7f086cd43fb8446aaae928443472e838, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/5b60071ef32b46abbd7f294650fa673f, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/efa0358c15bb4decae825b6625aa1c95, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/850deb7da4604dd2b983c0a1c5f7837f, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/9faa5da879f84c549e9fa3013c79be06] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp, totalSize=72.2 K 2023-03-20 10:57:39,704 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 49b9e7aae31245e3b151c438355eed89, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=1174, earliestPutTs=1679309855056 2023-03-20 10:57:39,704 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting e410618932b145a2803c2b40a7b1d850, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1210, earliestPutTs=1679309855179 2023-03-20 10:57:39,704 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 7f086cd43fb8446aaae928443472e838, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1210, earliestPutTs=1679309855179 2023-03-20 10:57:39,704 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting bad8240ce9894461bb7627f369a25a8a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1224, earliestPutTs=1679309855810 2023-03-20 10:57:39,705 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 5b60071ef32b46abbd7f294650fa673f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1224, earliestPutTs=1679309855810 2023-03-20 10:57:39,705 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 39d8474f08484aac92143096dcd5acf3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1260, earliestPutTs=1679309857056 2023-03-20 10:57:39,705 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting efa0358c15bb4decae825b6625aa1c95, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1260, earliestPutTs=1679309857056 2023-03-20 10:57:39,705 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting ef0713c23f0f4cf58f57214d4ae48989, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1271, earliestPutTs=1679309857618 2023-03-20 10:57:39,705 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 850deb7da4604dd2b983c0a1c5f7837f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1271, earliestPutTs=1679309857618 2023-03-20 10:57:39,706 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 16fcc391c6144109bafd8f18c94700c0, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=1281, earliestPutTs=1679309859202 2023-03-20 10:57:39,707 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 9faa5da879f84c549e9fa3013c79be06, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1281, earliestPutTs=1679309859214 2023-03-20 10:57:39,728 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] throttle.PressureAwareThroughputController(145): 72402f7d20e803d356627e208deb6fe2#A#compaction#282 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:57:39,733 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): 72402f7d20e803d356627e208deb6fe2#B#compaction#283 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:57:39,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 72402f7d20e803d356627e208deb6fe2 2023-03-20 10:57:39,776 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 72402f7d20e803d356627e208deb6fe2 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-03-20 10:57:39,776 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=A 2023-03-20 10:57:39,776 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-20 10:57:39,776 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=B 2023-03-20 10:57:39,776 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-20 10:57:39,776 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=C 2023-03-20 10:57:39,776 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-20 10:57:39,787 DEBUG [Thread-1564] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:39,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:39,789 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:39,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:39,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:39,790 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:39,790 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:39,790 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:39,790 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:39,790 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:39,790 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:39,791 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,791 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:39,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:39,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:39,791 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,791 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:39,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:39,791 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:39,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:39,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:39,792 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:39,792 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:39,792 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:39,792 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:39,792 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:39,792 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:39,792 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:39,793 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:39,793 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:39,793 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. started... 2023-03-20 10:57:39,793 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. as already flushing 2023-03-20 10:57:39,793 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:39,793 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:39,793 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:39,793 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,793 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:39,793 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:39,794 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:39,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:39,794 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:39,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:39,794 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:39,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:39,794 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:39,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:39,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:39,795 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:39,796 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72bfc9f1[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:39,796 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:39,796 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:39,796 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:39,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:39,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:39,796 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:39,796 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:39,796 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:39,796 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,796 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:39,797 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,797 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:39,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:39,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:39,797 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:39,797 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:39,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:39,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:39,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:39,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:39,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:39,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:39,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:39,799 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:39,799 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:39,799 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:39,800 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:39,800 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:39,800 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:39,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:39,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:39,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:39,800 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:39,801 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:39,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:39,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:39,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:39,801 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:39,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:39,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:39,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:39,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:39,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:39,802 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:39,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:39,803 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,803 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:39,803 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:39,803 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:39,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:39,803 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:39,803 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:39,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:39,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:39,803 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:39,803 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:39,803 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:39,803 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:39,803 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:57:39,803 DEBUG [Thread-1564] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:39,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:39,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:39,803 DEBUG [Thread-1564] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:39,808 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:39,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 424 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55168 deadline: 1679309919805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,835 DEBUG [Thread-1567] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7bae1a64 to 127.0.0.1:60226 2023-03-20 10:57:39,835 DEBUG [Thread-1565] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6b458e37 to 127.0.0.1:60226 2023-03-20 10:57:39,835 DEBUG [Thread-1567] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 10:57:39,835 DEBUG [Thread-1565] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 10:57:39,836 DEBUG [Thread-1573] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7e5e7a93 to 127.0.0.1:60226 2023-03-20 10:57:39,836 DEBUG [Thread-1573] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 10:57:39,838 DEBUG [Thread-1571] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x509adaa8 to 127.0.0.1:60226 2023-03-20 10:57:39,838 DEBUG [Thread-1571] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 10:57:39,838 DEBUG [Thread-1569] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5654f8ca to 127.0.0.1:60226 2023-03-20 10:57:39,838 DEBUG [Thread-1569] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 10:57:39,904 DEBUG [Thread-1564] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:39,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:39,910 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:39,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 426 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55168 deadline: 1679309919910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,981 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:39,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 417 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55136 deadline: 1679309919981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:39,987 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:39,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 355 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55182 deadline: 1679309919986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:40,112 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:40,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 428 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55168 deadline: 1679309920112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:40,160 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/A/7b1d3eb267ba40478287c4506d5e2c46 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/7b1d3eb267ba40478287c4506d5e2c46 2023-03-20 10:57:40,166 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 72402f7d20e803d356627e208deb6fe2/A of 72402f7d20e803d356627e208deb6fe2 into 7b1d3eb267ba40478287c4506d5e2c46(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:57:40,166 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 72402f7d20e803d356627e208deb6fe2: 2023-03-20 10:57:40,166 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2., storeName=72402f7d20e803d356627e208deb6fe2/A, priority=10, startTime=1679309859698; duration=0sec 2023-03-20 10:57:40,166 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:57:40,166 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-20 10:57:40,169 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73934 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-20 10:57:40,169 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1912): 72402f7d20e803d356627e208deb6fe2/C is initiating minor compaction (all files) 2023-03-20 10:57:40,169 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 72402f7d20e803d356627e208deb6fe2/C in TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:40,170 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/956a1308fd7a4dadbc1c71ea93415005, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/fc56513ac898486eb93451797490f7c6, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/621afc1df2e94efb88c98f0f6814fe22, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/ed51c9e3b10442ce9955af7de79a3232, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/2420f8e532e843aabee79e586d364b59, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/b836f7736c9c4af6831ca8fedbea3b2c] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp, totalSize=72.2 K 2023-03-20 10:57:40,170 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 956a1308fd7a4dadbc1c71ea93415005, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=1174, earliestPutTs=1679309855056 2023-03-20 10:57:40,171 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting fc56513ac898486eb93451797490f7c6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1210, earliestPutTs=1679309855179 2023-03-20 10:57:40,171 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 621afc1df2e94efb88c98f0f6814fe22, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1224, earliestPutTs=1679309855810 2023-03-20 10:57:40,172 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting ed51c9e3b10442ce9955af7de79a3232, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1260, earliestPutTs=1679309857056 2023-03-20 10:57:40,172 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 2420f8e532e843aabee79e586d364b59, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1271, earliestPutTs=1679309857618 2023-03-20 10:57:40,172 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting b836f7736c9c4af6831ca8fedbea3b2c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1281, earliestPutTs=1679309859214 2023-03-20 10:57:40,172 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/B/c7a504a0149241b4994941acaa720e09 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/c7a504a0149241b4994941acaa720e09 2023-03-20 10:57:40,177 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 72402f7d20e803d356627e208deb6fe2/B of 72402f7d20e803d356627e208deb6fe2 into c7a504a0149241b4994941acaa720e09(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:57:40,177 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 72402f7d20e803d356627e208deb6fe2: 2023-03-20 10:57:40,177 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2., storeName=72402f7d20e803d356627e208deb6fe2/B, priority=10, startTime=1679309859699; duration=0sec 2023-03-20 10:57:40,177 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:57:40,181 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] throttle.PressureAwareThroughputController(145): 72402f7d20e803d356627e208deb6fe2#C#compaction#285 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:57:40,187 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=1319 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/A/e26cbcbd7c944fc3adbff551924b502d 2023-03-20 10:57:40,418 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:40,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 430 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55168 deadline: 1679309920418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:40,484 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:40,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 419 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55136 deadline: 1679309920484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:40,490 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:40,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 357 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55182 deadline: 1679309920490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:40,589 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/C/469e24de9fc14604ac32c54a0665737e as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/469e24de9fc14604ac32c54a0665737e 2023-03-20 10:57:40,592 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 72402f7d20e803d356627e208deb6fe2/C of 72402f7d20e803d356627e208deb6fe2 into 469e24de9fc14604ac32c54a0665737e(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:57:40,593 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 72402f7d20e803d356627e208deb6fe2: 2023-03-20 10:57:40,593 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2., storeName=72402f7d20e803d356627e208deb6fe2/C, priority=10, startTime=1679309859699; duration=0sec 2023-03-20 10:57:40,593 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:57:40,600 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=1319 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/B/538d5e11d49440be9efd802c4fa78269 2023-03-20 10:57:40,921 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:40,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 432 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:55168 deadline: 1679309920921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:41,009 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=1319 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/C/1e2c5ae50bb342399f1b4f76bd8b9254 2023-03-20 10:57:41,013 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/A/e26cbcbd7c944fc3adbff551924b502d as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/e26cbcbd7c944fc3adbff551924b502d 2023-03-20 10:57:41,016 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/e26cbcbd7c944fc3adbff551924b502d, entries=200, sequenceid=1319, filesize=14.3 K 2023-03-20 10:57:41,017 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/B/538d5e11d49440be9efd802c4fa78269 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/538d5e11d49440be9efd802c4fa78269 2023-03-20 10:57:41,020 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/538d5e11d49440be9efd802c4fa78269, entries=150, sequenceid=1319, filesize=11.9 K 2023-03-20 10:57:41,021 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/C/1e2c5ae50bb342399f1b4f76bd8b9254 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/1e2c5ae50bb342399f1b4f76bd8b9254 2023-03-20 10:57:41,025 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/1e2c5ae50bb342399f1b4f76bd8b9254, entries=150, sequenceid=1319, filesize=11.9 K 2023-03-20 10:57:41,026 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~405.56 KB/415296, currentSize=40.25 KB/41220 for 72402f7d20e803d356627e208deb6fe2 in 1250ms, sequenceid=1319, compaction requested=false 2023-03-20 10:57:41,026 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 72402f7d20e803d356627e208deb6fe2: 2023-03-20 10:57:41,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 72402f7d20e803d356627e208deb6fe2 2023-03-20 10:57:41,489 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 72402f7d20e803d356627e208deb6fe2 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-20 10:57:41,489 DEBUG [Thread-1562] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x62be7a0b to 127.0.0.1:60226 2023-03-20 10:57:41,489 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=A 2023-03-20 10:57:41,489 DEBUG [Thread-1562] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 10:57:41,489 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:41,489 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=B 2023-03-20 10:57:41,490 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:41,490 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=C 2023-03-20 10:57:41,490 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:41,498 DEBUG [Thread-1556] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x712d78ca to 127.0.0.1:60226 2023-03-20 10:57:41,498 DEBUG [Thread-1556] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 10:57:41,904 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=1332 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/A/b3bcbf64fe824614a91493397c6dcd3b 2023-03-20 10:57:41,932 DEBUG [Thread-1560] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7f456fa2 to 127.0.0.1:60226 2023-03-20 10:57:41,932 DEBUG [Thread-1560] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 10:57:42,354 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=1332 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/B/efcd0fb2b39248b1b260715aef6f4f8b 2023-03-20 10:57:42,364 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=1332 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/C/1b6b0fef38674e1cac81b93dffed1327 2023-03-20 10:57:42,367 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/A/b3bcbf64fe824614a91493397c6dcd3b as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/b3bcbf64fe824614a91493397c6dcd3b 2023-03-20 10:57:42,372 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/b3bcbf64fe824614a91493397c6dcd3b, entries=150, sequenceid=1332, filesize=11.9 K 2023-03-20 10:57:42,374 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/B/efcd0fb2b39248b1b260715aef6f4f8b as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/efcd0fb2b39248b1b260715aef6f4f8b 2023-03-20 10:57:42,383 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/efcd0fb2b39248b1b260715aef6f4f8b, entries=150, sequenceid=1332, filesize=11.9 K 2023-03-20 10:57:42,384 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/C/1b6b0fef38674e1cac81b93dffed1327 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/1b6b0fef38674e1cac81b93dffed1327 2023-03-20 10:57:42,392 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/1b6b0fef38674e1cac81b93dffed1327, entries=150, sequenceid=1332, filesize=11.9 K 2023-03-20 10:57:42,393 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=13.42 KB/13740 for 72402f7d20e803d356627e208deb6fe2 in 904ms, sequenceid=1332, compaction requested=true 2023-03-20 10:57:42,393 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 72402f7d20e803d356627e208deb6fe2: 2023-03-20 10:57:42,393 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:57:42,393 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 10:57:42,393 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 10:57:42,394 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:57:42,394 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:57:42,402 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37580 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 10:57:42,402 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40020 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 10:57:42,403 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): 72402f7d20e803d356627e208deb6fe2/B is initiating minor compaction (all files) 2023-03-20 10:57:42,403 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1912): 72402f7d20e803d356627e208deb6fe2/A is initiating minor compaction (all files) 2023-03-20 10:57:42,403 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 72402f7d20e803d356627e208deb6fe2/B in TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:42,403 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 72402f7d20e803d356627e208deb6fe2/A in TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:42,403 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/c7a504a0149241b4994941acaa720e09, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/538d5e11d49440be9efd802c4fa78269, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/efcd0fb2b39248b1b260715aef6f4f8b] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp, totalSize=36.7 K 2023-03-20 10:57:42,403 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/7b1d3eb267ba40478287c4506d5e2c46, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/e26cbcbd7c944fc3adbff551924b502d, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/b3bcbf64fe824614a91493397c6dcd3b] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp, totalSize=39.1 K 2023-03-20 10:57:42,403 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting c7a504a0149241b4994941acaa720e09, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=1281, earliestPutTs=1679309859214 2023-03-20 10:57:42,403 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 7b1d3eb267ba40478287c4506d5e2c46, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=1281, earliestPutTs=1679309859214 2023-03-20 10:57:42,403 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 538d5e11d49440be9efd802c4fa78269, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1319, earliestPutTs=1679309859359 2023-03-20 10:57:42,403 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting e26cbcbd7c944fc3adbff551924b502d, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=1319, earliestPutTs=1679309859359 2023-03-20 10:57:42,404 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting efcd0fb2b39248b1b260715aef6f4f8b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1332, earliestPutTs=1679309859796 2023-03-20 10:57:42,404 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting b3bcbf64fe824614a91493397c6dcd3b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1332, earliestPutTs=1679309859796 2023-03-20 10:57:42,416 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] throttle.PressureAwareThroughputController(145): 72402f7d20e803d356627e208deb6fe2#A#compaction#291 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:57:42,417 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): 72402f7d20e803d356627e208deb6fe2#B#compaction#292 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:57:42,829 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/A/1c72e53945b641a287b3d4d537a735bf as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/1c72e53945b641a287b3d4d537a735bf 2023-03-20 10:57:42,829 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/B/5fc6ef2e56d74858945605198b5c770b as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/5fc6ef2e56d74858945605198b5c770b 2023-03-20 10:57:42,833 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 72402f7d20e803d356627e208deb6fe2/A of 72402f7d20e803d356627e208deb6fe2 into 1c72e53945b641a287b3d4d537a735bf(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:57:42,833 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 72402f7d20e803d356627e208deb6fe2: 2023-03-20 10:57:42,833 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 72402f7d20e803d356627e208deb6fe2/B of 72402f7d20e803d356627e208deb6fe2 into 5fc6ef2e56d74858945605198b5c770b(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:57:42,833 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2., storeName=72402f7d20e803d356627e208deb6fe2/A, priority=13, startTime=1679309862393; duration=0sec 2023-03-20 10:57:42,833 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 72402f7d20e803d356627e208deb6fe2: 2023-03-20 10:57:42,833 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2., storeName=72402f7d20e803d356627e208deb6fe2/B, priority=13, startTime=1679309862393; duration=0sec 2023-03-20 10:57:42,834 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:57:42,834 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:57:42,834 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 10:57:42,834 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37580 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 10:57:42,835 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1912): 72402f7d20e803d356627e208deb6fe2/C is initiating minor compaction (all files) 2023-03-20 10:57:42,835 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 72402f7d20e803d356627e208deb6fe2/C in TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:42,835 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/469e24de9fc14604ac32c54a0665737e, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/1e2c5ae50bb342399f1b4f76bd8b9254, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/1b6b0fef38674e1cac81b93dffed1327] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp, totalSize=36.7 K 2023-03-20 10:57:42,835 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 469e24de9fc14604ac32c54a0665737e, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=1281, earliestPutTs=1679309859214 2023-03-20 10:57:42,836 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 1e2c5ae50bb342399f1b4f76bd8b9254, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1319, earliestPutTs=1679309859359 2023-03-20 10:57:42,836 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 1b6b0fef38674e1cac81b93dffed1327, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1332, earliestPutTs=1679309859796 2023-03-20 10:57:42,844 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] throttle.PressureAwareThroughputController(145): 72402f7d20e803d356627e208deb6fe2#C#compaction#293 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:57:43,263 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/C/d6c2f00470d848f8a88825ddff07c238 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/d6c2f00470d848f8a88825ddff07c238 2023-03-20 10:57:43,267 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 72402f7d20e803d356627e208deb6fe2/C of 72402f7d20e803d356627e208deb6fe2 into d6c2f00470d848f8a88825ddff07c238(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:57:43,267 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 72402f7d20e803d356627e208deb6fe2: 2023-03-20 10:57:43,267 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2., storeName=72402f7d20e803d356627e208deb6fe2/C, priority=13, startTime=1679309862394; duration=0sec 2023-03-20 10:57:43,267 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:57:43,389 DEBUG [Thread-1558] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x407b9cb0 to 127.0.0.1:60226 2023-03-20 10:57:43,389 DEBUG [Thread-1558] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 10:57:44,009 DEBUG [Thread-1554] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7187fcf3 to 127.0.0.1:60226 2023-03-20 10:57:44,009 DEBUG [Thread-1554] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 10:57:44,009 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-20 10:57:44,009 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(394): wrote 194 2023-03-20 10:57:44,009 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(394): wrote 236 2023-03-20 10:57:44,009 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(394): wrote 227 2023-03-20 10:57:44,009 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(394): wrote 295 2023-03-20 10:57:44,009 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(394): wrote 284 2023-03-20 10:57:44,009 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-20 10:57:44,009 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(398): read 3638 2023-03-20 10:57:44,009 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(398): read 3628 2023-03-20 10:57:44,010 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(398): read 3478 2023-03-20 10:57:44,010 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(398): read 3679 2023-03-20 10:57:44,010 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(398): read 3674 2023-03-20 10:57:44,010 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-20 10:57:44,010 INFO [Listener at localhost.localdomain/43509] client.ConnectionImplementation(1959): Closing master protocol: MasterService 2023-03-20 10:57:44,010 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4d3f4b11 to 127.0.0.1:60226 2023-03-20 10:57:44,010 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 10:57:44,011 INFO [Listener at localhost.localdomain/43509] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-20 10:57:44,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-03-20 10:57:44,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure2.ProcedureExecutor(1029): Stored pid=31, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-20 10:57:44,016 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679309864016"}]},"ts":"1679309864016"} 2023-03-20 10:57:44,017 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-20 10:57:44,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-20 10:57:44,113 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-20 10:57:44,114 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=72402f7d20e803d356627e208deb6fe2, UNASSIGN}] 2023-03-20 10:57:44,115 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=72402f7d20e803d356627e208deb6fe2, UNASSIGN 2023-03-20 10:57:44,116 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=72402f7d20e803d356627e208deb6fe2, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:44,116 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679309864116"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679309864116"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679309864116"}]},"ts":"1679309864116"} 2023-03-20 10:57:44,117 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=32, state=RUNNABLE; CloseRegionProcedure 72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062}] 2023-03-20 10:57:44,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-20 10:57:44,269 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 72402f7d20e803d356627e208deb6fe2 2023-03-20 10:57:44,270 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 72402f7d20e803d356627e208deb6fe2, disabling compactions & flushes 2023-03-20 10:57:44,270 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:44,270 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:44,270 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. after waiting 0 ms 2023-03-20 10:57:44,270 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:44,270 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 72402f7d20e803d356627e208deb6fe2 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-20 10:57:44,270 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=A 2023-03-20 10:57:44,270 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:44,270 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=B 2023-03-20 10:57:44,270 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:44,270 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 72402f7d20e803d356627e208deb6fe2, store=C 2023-03-20 10:57:44,270 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:44,277 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1342 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/A/31c06de0cb36493c88a1ab5ed833e72d 2023-03-20 10:57:44,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-20 10:57:44,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-20 10:57:44,688 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1342 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/B/4253299e165b479ea4649df65f0ce1b1 2023-03-20 10:57:45,117 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1342 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/C/d99204e6541c4d2ca14277dd0c9032e9 2023-03-20 10:57:45,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-20 10:57:45,123 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/A/31c06de0cb36493c88a1ab5ed833e72d as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/31c06de0cb36493c88a1ab5ed833e72d 2023-03-20 10:57:45,128 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/31c06de0cb36493c88a1ab5ed833e72d, entries=100, sequenceid=1342, filesize=9.5 K 2023-03-20 10:57:45,129 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/B/4253299e165b479ea4649df65f0ce1b1 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/4253299e165b479ea4649df65f0ce1b1 2023-03-20 10:57:45,136 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/4253299e165b479ea4649df65f0ce1b1, entries=100, sequenceid=1342, filesize=9.5 K 2023-03-20 10:57:45,137 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/.tmp/C/d99204e6541c4d2ca14277dd0c9032e9 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/d99204e6541c4d2ca14277dd0c9032e9 2023-03-20 10:57:45,143 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/d99204e6541c4d2ca14277dd0c9032e9, entries=100, sequenceid=1342, filesize=9.5 K 2023-03-20 10:57:45,144 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 72402f7d20e803d356627e208deb6fe2 in 874ms, sequenceid=1342, compaction requested=false 2023-03-20 10:57:45,164 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/e0f2074420494f18892dd06834b239df, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/edc884406ce34f9f9441ff0cd73956ba, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/0d344dfab12d46749d37c0a4e6b84600, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/4f8b9b806fb64895b4075ab6dfdf4928, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/807d4f3ae4514bb8b8b91ff8e52a7776, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/612f5f5d981543779183450ac9c6e51b, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/6c802ff1c7604d9fb67fa2d19a4e97e5, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/7be9cb127c0a4694b69cf6619b74e3b7, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/626d86ecae6d40948450d611bc3cca4e, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/351daccce60f467a92e3f352b65ad8e1, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/5cc3fd7208e04bfba831f54248749910, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/0e3c87e90103476899411b351fe0c4d0, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/82caf51f9cf24f05873667083f6ad1b2, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/e2a8475cccf44a0c98fd75806af54c65, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/7da10cda8de3491b9c7b236e9a248bd7, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/bcc1226eb9f1486b9a920fd091ec88c6, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/b146f4acfc4f4bcd928a9ed18b614e17, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/f9456bb1ce61475f84976af7da9a70f1, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/db92ab837db84ff4a34f63c1483ad329, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/24c544bc9cf942a08b5f9c6fe7d59bc6, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/dfa4dc201306418fb87832a9dc8d16e0, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/094f7551839f4b0d908394149b12a713, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/5db7f6101fa245ef8a962090dfd2c69b, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/71cfb0027f744c52a2e1d4f24e14b524, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/c2a3e332bff4422b940636c87dc92247, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/e410618932b145a2803c2b40a7b1d850, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/bad8240ce9894461bb7627f369a25a8a, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/39d8474f08484aac92143096dcd5acf3, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/ef0713c23f0f4cf58f57214d4ae48989, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/16fcc391c6144109bafd8f18c94700c0, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/7b1d3eb267ba40478287c4506d5e2c46, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/e26cbcbd7c944fc3adbff551924b502d, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/b3bcbf64fe824614a91493397c6dcd3b] to archive 2023-03-20 10:57:45,165 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-20 10:57:45,167 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/e0f2074420494f18892dd06834b239df to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/e0f2074420494f18892dd06834b239df 2023-03-20 10:57:45,168 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/edc884406ce34f9f9441ff0cd73956ba to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/edc884406ce34f9f9441ff0cd73956ba 2023-03-20 10:57:45,169 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/0d344dfab12d46749d37c0a4e6b84600 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/0d344dfab12d46749d37c0a4e6b84600 2023-03-20 10:57:45,170 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/4f8b9b806fb64895b4075ab6dfdf4928 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/4f8b9b806fb64895b4075ab6dfdf4928 2023-03-20 10:57:45,171 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/807d4f3ae4514bb8b8b91ff8e52a7776 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/807d4f3ae4514bb8b8b91ff8e52a7776 2023-03-20 10:57:45,172 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/612f5f5d981543779183450ac9c6e51b to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/612f5f5d981543779183450ac9c6e51b 2023-03-20 10:57:45,173 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/6c802ff1c7604d9fb67fa2d19a4e97e5 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/6c802ff1c7604d9fb67fa2d19a4e97e5 2023-03-20 10:57:45,174 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/7be9cb127c0a4694b69cf6619b74e3b7 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/7be9cb127c0a4694b69cf6619b74e3b7 2023-03-20 10:57:45,174 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/626d86ecae6d40948450d611bc3cca4e to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/626d86ecae6d40948450d611bc3cca4e 2023-03-20 10:57:45,175 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/351daccce60f467a92e3f352b65ad8e1 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/351daccce60f467a92e3f352b65ad8e1 2023-03-20 10:57:45,176 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/5cc3fd7208e04bfba831f54248749910 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/5cc3fd7208e04bfba831f54248749910 2023-03-20 10:57:45,177 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/0e3c87e90103476899411b351fe0c4d0 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/0e3c87e90103476899411b351fe0c4d0 2023-03-20 10:57:45,178 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/82caf51f9cf24f05873667083f6ad1b2 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/82caf51f9cf24f05873667083f6ad1b2 2023-03-20 10:57:45,180 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/e2a8475cccf44a0c98fd75806af54c65 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/e2a8475cccf44a0c98fd75806af54c65 2023-03-20 10:57:45,181 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/7da10cda8de3491b9c7b236e9a248bd7 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/7da10cda8de3491b9c7b236e9a248bd7 2023-03-20 10:57:45,182 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/bcc1226eb9f1486b9a920fd091ec88c6 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/bcc1226eb9f1486b9a920fd091ec88c6 2023-03-20 10:57:45,183 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/b146f4acfc4f4bcd928a9ed18b614e17 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/b146f4acfc4f4bcd928a9ed18b614e17 2023-03-20 10:57:45,184 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/f9456bb1ce61475f84976af7da9a70f1 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/f9456bb1ce61475f84976af7da9a70f1 2023-03-20 10:57:45,185 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/db92ab837db84ff4a34f63c1483ad329 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/db92ab837db84ff4a34f63c1483ad329 2023-03-20 10:57:45,187 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/24c544bc9cf942a08b5f9c6fe7d59bc6 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/24c544bc9cf942a08b5f9c6fe7d59bc6 2023-03-20 10:57:45,188 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/dfa4dc201306418fb87832a9dc8d16e0 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/dfa4dc201306418fb87832a9dc8d16e0 2023-03-20 10:57:45,189 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/094f7551839f4b0d908394149b12a713 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/094f7551839f4b0d908394149b12a713 2023-03-20 10:57:45,190 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/5db7f6101fa245ef8a962090dfd2c69b to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/5db7f6101fa245ef8a962090dfd2c69b 2023-03-20 10:57:45,191 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/71cfb0027f744c52a2e1d4f24e14b524 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/71cfb0027f744c52a2e1d4f24e14b524 2023-03-20 10:57:45,192 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/c2a3e332bff4422b940636c87dc92247 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/c2a3e332bff4422b940636c87dc92247 2023-03-20 10:57:45,194 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/e410618932b145a2803c2b40a7b1d850 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/e410618932b145a2803c2b40a7b1d850 2023-03-20 10:57:45,195 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/bad8240ce9894461bb7627f369a25a8a to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/bad8240ce9894461bb7627f369a25a8a 2023-03-20 10:57:45,196 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/39d8474f08484aac92143096dcd5acf3 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/39d8474f08484aac92143096dcd5acf3 2023-03-20 10:57:45,197 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/ef0713c23f0f4cf58f57214d4ae48989 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/ef0713c23f0f4cf58f57214d4ae48989 2023-03-20 10:57:45,198 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/16fcc391c6144109bafd8f18c94700c0 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/16fcc391c6144109bafd8f18c94700c0 2023-03-20 10:57:45,199 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/7b1d3eb267ba40478287c4506d5e2c46 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/7b1d3eb267ba40478287c4506d5e2c46 2023-03-20 10:57:45,200 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/e26cbcbd7c944fc3adbff551924b502d to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/e26cbcbd7c944fc3adbff551924b502d 2023-03-20 10:57:45,201 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/b3bcbf64fe824614a91493397c6dcd3b to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/b3bcbf64fe824614a91493397c6dcd3b 2023-03-20 10:57:45,205 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/fc0188b7d1fa415fb217910da8214522, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/267a75d458d944a9a9304cef4a9881cb, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/bc6fd36293f34956a497b5a66c4f7507, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/5513e593024f49da943e0691c383825c, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/e967d0bedf4a47789400ecc5dd7c2ad2, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/ecbb1475a4fe40779070fd1a43948990, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/6e0a2c2ace774dafb0c2812476d863f2, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/73f655f32858409c9963dd40947c8de3, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/a38aa04d1ad64cdb988405db45030ffc, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/2a7041c802ed4d5d98d08ecc9ad0224d, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/f2926a57fb7942afb76bbe8e8db0b8d9, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/9f154e8af2b64c4b945607fe41e00e82, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/47dc95c30fae441aa87572d9b2ac4340, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/70cd43003c784875887fd459af37f552, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/a379fa2d36f44455be9902dc6e1d3e46, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/435b40fb08be42e4b58aa72f9ca6d775, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/b0145fca1319491bbf81eb70b6dddfde, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/0ca593230ae54512968e6a7b5bf2c0b9, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/ecba251bf7194fa0bc095368ddf3c6fd, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/cf227877aeac411bbfd455262e019b35, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/cbe2753a2ab942ea978c5de5485651bc, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/627bec0973d14a9d86b0d47559ab6973, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/cfb92fb4773c46d5822ee1dd2471037a, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/49b9e7aae31245e3b151c438355eed89, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/0ca86140ea82412080f4f0ca5a59c862, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/7f086cd43fb8446aaae928443472e838, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/5b60071ef32b46abbd7f294650fa673f, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/efa0358c15bb4decae825b6625aa1c95, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/850deb7da4604dd2b983c0a1c5f7837f, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/c7a504a0149241b4994941acaa720e09, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/9faa5da879f84c549e9fa3013c79be06, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/538d5e11d49440be9efd802c4fa78269, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/efcd0fb2b39248b1b260715aef6f4f8b] to archive 2023-03-20 10:57:45,206 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-20 10:57:45,208 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/fc0188b7d1fa415fb217910da8214522 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/fc0188b7d1fa415fb217910da8214522 2023-03-20 10:57:45,209 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/267a75d458d944a9a9304cef4a9881cb to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/267a75d458d944a9a9304cef4a9881cb 2023-03-20 10:57:45,210 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/bc6fd36293f34956a497b5a66c4f7507 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/bc6fd36293f34956a497b5a66c4f7507 2023-03-20 10:57:45,211 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/5513e593024f49da943e0691c383825c to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/5513e593024f49da943e0691c383825c 2023-03-20 10:57:45,212 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/e967d0bedf4a47789400ecc5dd7c2ad2 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/e967d0bedf4a47789400ecc5dd7c2ad2 2023-03-20 10:57:45,212 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/ecbb1475a4fe40779070fd1a43948990 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/ecbb1475a4fe40779070fd1a43948990 2023-03-20 10:57:45,213 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/6e0a2c2ace774dafb0c2812476d863f2 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/6e0a2c2ace774dafb0c2812476d863f2 2023-03-20 10:57:45,214 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/73f655f32858409c9963dd40947c8de3 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/73f655f32858409c9963dd40947c8de3 2023-03-20 10:57:45,214 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/a38aa04d1ad64cdb988405db45030ffc to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/a38aa04d1ad64cdb988405db45030ffc 2023-03-20 10:57:45,215 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/2a7041c802ed4d5d98d08ecc9ad0224d to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/2a7041c802ed4d5d98d08ecc9ad0224d 2023-03-20 10:57:45,216 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/f2926a57fb7942afb76bbe8e8db0b8d9 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/f2926a57fb7942afb76bbe8e8db0b8d9 2023-03-20 10:57:45,217 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/9f154e8af2b64c4b945607fe41e00e82 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/9f154e8af2b64c4b945607fe41e00e82 2023-03-20 10:57:45,218 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/47dc95c30fae441aa87572d9b2ac4340 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/47dc95c30fae441aa87572d9b2ac4340 2023-03-20 10:57:45,219 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/70cd43003c784875887fd459af37f552 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/70cd43003c784875887fd459af37f552 2023-03-20 10:57:45,219 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/a379fa2d36f44455be9902dc6e1d3e46 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/a379fa2d36f44455be9902dc6e1d3e46 2023-03-20 10:57:45,220 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/435b40fb08be42e4b58aa72f9ca6d775 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/435b40fb08be42e4b58aa72f9ca6d775 2023-03-20 10:57:45,221 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/b0145fca1319491bbf81eb70b6dddfde to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/b0145fca1319491bbf81eb70b6dddfde 2023-03-20 10:57:45,222 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/0ca593230ae54512968e6a7b5bf2c0b9 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/0ca593230ae54512968e6a7b5bf2c0b9 2023-03-20 10:57:45,223 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/ecba251bf7194fa0bc095368ddf3c6fd to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/ecba251bf7194fa0bc095368ddf3c6fd 2023-03-20 10:57:45,224 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/cf227877aeac411bbfd455262e019b35 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/cf227877aeac411bbfd455262e019b35 2023-03-20 10:57:45,225 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/cbe2753a2ab942ea978c5de5485651bc to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/cbe2753a2ab942ea978c5de5485651bc 2023-03-20 10:57:45,225 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/627bec0973d14a9d86b0d47559ab6973 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/627bec0973d14a9d86b0d47559ab6973 2023-03-20 10:57:45,226 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/cfb92fb4773c46d5822ee1dd2471037a to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/cfb92fb4773c46d5822ee1dd2471037a 2023-03-20 10:57:45,227 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/49b9e7aae31245e3b151c438355eed89 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/49b9e7aae31245e3b151c438355eed89 2023-03-20 10:57:45,228 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/0ca86140ea82412080f4f0ca5a59c862 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/0ca86140ea82412080f4f0ca5a59c862 2023-03-20 10:57:45,229 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/7f086cd43fb8446aaae928443472e838 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/7f086cd43fb8446aaae928443472e838 2023-03-20 10:57:45,230 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/5b60071ef32b46abbd7f294650fa673f to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/5b60071ef32b46abbd7f294650fa673f 2023-03-20 10:57:45,232 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/efa0358c15bb4decae825b6625aa1c95 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/efa0358c15bb4decae825b6625aa1c95 2023-03-20 10:57:45,233 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/850deb7da4604dd2b983c0a1c5f7837f to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/850deb7da4604dd2b983c0a1c5f7837f 2023-03-20 10:57:45,234 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/c7a504a0149241b4994941acaa720e09 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/c7a504a0149241b4994941acaa720e09 2023-03-20 10:57:45,235 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/9faa5da879f84c549e9fa3013c79be06 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/9faa5da879f84c549e9fa3013c79be06 2023-03-20 10:57:45,236 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/538d5e11d49440be9efd802c4fa78269 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/538d5e11d49440be9efd802c4fa78269 2023-03-20 10:57:45,237 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/efcd0fb2b39248b1b260715aef6f4f8b to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/efcd0fb2b39248b1b260715aef6f4f8b 2023-03-20 10:57:45,242 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/0de5c1a4660d403aaa2ca6c445f40991, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/f88cb61c68f646cbbf1eea0284a24779, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/40b1a0810c8e4ee3a04c8cdb6c180d49, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/40e5c692a5cd4e958aec77a681dbe2da, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/5a1be9933de24bc4acf224f5fd026dbd, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/e1a7d61b163b46ed8c107c37f9b782d8, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/9ed9e7f6299044d9b88f30fcfd59890d, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/a0883a66603149ebb7cf6a4b28270c60, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/fef22325c3594bcab54e13d01f900e2c, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/e18ee315a83546bb984a40b61e2a1158, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/612db19d8c00481fb117c2a2ec284186, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/152e471c83f349038479f52c0c56df19, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/fbb612ed572d4ec78f167e43b8487fbe, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/d9a2b25c8fcb4182a6440333e3166d25, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/dc5782915282432d9cc78dda1b2a9b45, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/cbe2d2a5562841edb86ec55ab4ddc298, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/1d3848cec2e541259d2816cf2fd28d80, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/8d010a3e3b4a4a43bba443e353262666, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/da0c3fc6857243809f06c1bbf536d0b0, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/9fd8492d5bdd4efb8d9d5eed33aeac40, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/e63905300ba24d3db5312a6f7d9aa25c, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/a91124e5d5c74ca9b57d3d4308daf4bd, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/18ec1c8a2bef437bac8fcb310f1210d7, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/956a1308fd7a4dadbc1c71ea93415005, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/231b62a970164153a414d4b00a76d059, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/fc56513ac898486eb93451797490f7c6, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/621afc1df2e94efb88c98f0f6814fe22, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/ed51c9e3b10442ce9955af7de79a3232, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/2420f8e532e843aabee79e586d364b59, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/469e24de9fc14604ac32c54a0665737e, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/b836f7736c9c4af6831ca8fedbea3b2c, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/1e2c5ae50bb342399f1b4f76bd8b9254, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/1b6b0fef38674e1cac81b93dffed1327] to archive 2023-03-20 10:57:45,242 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-20 10:57:45,244 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/0de5c1a4660d403aaa2ca6c445f40991 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/0de5c1a4660d403aaa2ca6c445f40991 2023-03-20 10:57:45,245 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/f88cb61c68f646cbbf1eea0284a24779 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/f88cb61c68f646cbbf1eea0284a24779 2023-03-20 10:57:45,246 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/40b1a0810c8e4ee3a04c8cdb6c180d49 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/40b1a0810c8e4ee3a04c8cdb6c180d49 2023-03-20 10:57:45,247 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/40e5c692a5cd4e958aec77a681dbe2da to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/40e5c692a5cd4e958aec77a681dbe2da 2023-03-20 10:57:45,248 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/5a1be9933de24bc4acf224f5fd026dbd to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/5a1be9933de24bc4acf224f5fd026dbd 2023-03-20 10:57:45,249 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/e1a7d61b163b46ed8c107c37f9b782d8 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/e1a7d61b163b46ed8c107c37f9b782d8 2023-03-20 10:57:45,250 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/9ed9e7f6299044d9b88f30fcfd59890d to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/9ed9e7f6299044d9b88f30fcfd59890d 2023-03-20 10:57:45,251 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/a0883a66603149ebb7cf6a4b28270c60 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/a0883a66603149ebb7cf6a4b28270c60 2023-03-20 10:57:45,252 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/fef22325c3594bcab54e13d01f900e2c to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/fef22325c3594bcab54e13d01f900e2c 2023-03-20 10:57:45,253 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/e18ee315a83546bb984a40b61e2a1158 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/e18ee315a83546bb984a40b61e2a1158 2023-03-20 10:57:45,254 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/612db19d8c00481fb117c2a2ec284186 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/612db19d8c00481fb117c2a2ec284186 2023-03-20 10:57:45,255 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/152e471c83f349038479f52c0c56df19 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/152e471c83f349038479f52c0c56df19 2023-03-20 10:57:45,256 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/fbb612ed572d4ec78f167e43b8487fbe to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/fbb612ed572d4ec78f167e43b8487fbe 2023-03-20 10:57:45,258 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/d9a2b25c8fcb4182a6440333e3166d25 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/d9a2b25c8fcb4182a6440333e3166d25 2023-03-20 10:57:45,259 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/dc5782915282432d9cc78dda1b2a9b45 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/dc5782915282432d9cc78dda1b2a9b45 2023-03-20 10:57:45,260 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/cbe2d2a5562841edb86ec55ab4ddc298 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/cbe2d2a5562841edb86ec55ab4ddc298 2023-03-20 10:57:45,262 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/1d3848cec2e541259d2816cf2fd28d80 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/1d3848cec2e541259d2816cf2fd28d80 2023-03-20 10:57:45,263 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/8d010a3e3b4a4a43bba443e353262666 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/8d010a3e3b4a4a43bba443e353262666 2023-03-20 10:57:45,263 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/da0c3fc6857243809f06c1bbf536d0b0 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/da0c3fc6857243809f06c1bbf536d0b0 2023-03-20 10:57:45,264 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/9fd8492d5bdd4efb8d9d5eed33aeac40 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/9fd8492d5bdd4efb8d9d5eed33aeac40 2023-03-20 10:57:45,265 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/e63905300ba24d3db5312a6f7d9aa25c to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/e63905300ba24d3db5312a6f7d9aa25c 2023-03-20 10:57:45,266 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/a91124e5d5c74ca9b57d3d4308daf4bd to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/a91124e5d5c74ca9b57d3d4308daf4bd 2023-03-20 10:57:45,266 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/18ec1c8a2bef437bac8fcb310f1210d7 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/18ec1c8a2bef437bac8fcb310f1210d7 2023-03-20 10:57:45,267 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/956a1308fd7a4dadbc1c71ea93415005 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/956a1308fd7a4dadbc1c71ea93415005 2023-03-20 10:57:45,268 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/231b62a970164153a414d4b00a76d059 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/231b62a970164153a414d4b00a76d059 2023-03-20 10:57:45,268 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/fc56513ac898486eb93451797490f7c6 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/fc56513ac898486eb93451797490f7c6 2023-03-20 10:57:45,269 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/621afc1df2e94efb88c98f0f6814fe22 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/621afc1df2e94efb88c98f0f6814fe22 2023-03-20 10:57:45,269 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/ed51c9e3b10442ce9955af7de79a3232 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/ed51c9e3b10442ce9955af7de79a3232 2023-03-20 10:57:45,270 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/2420f8e532e843aabee79e586d364b59 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/2420f8e532e843aabee79e586d364b59 2023-03-20 10:57:45,271 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/469e24de9fc14604ac32c54a0665737e to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/469e24de9fc14604ac32c54a0665737e 2023-03-20 10:57:45,271 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/b836f7736c9c4af6831ca8fedbea3b2c to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/b836f7736c9c4af6831ca8fedbea3b2c 2023-03-20 10:57:45,272 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/1e2c5ae50bb342399f1b4f76bd8b9254 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/1e2c5ae50bb342399f1b4f76bd8b9254 2023-03-20 10:57:45,273 DEBUG [StoreCloser-TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/1b6b0fef38674e1cac81b93dffed1327 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/1b6b0fef38674e1cac81b93dffed1327 2023-03-20 10:57:45,276 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/recovered.edits/1345.seqid, newMaxSeqId=1345, maxSeqId=1 2023-03-20 10:57:45,277 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2. 2023-03-20 10:57:45,277 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 72402f7d20e803d356627e208deb6fe2: 2023-03-20 10:57:45,278 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 72402f7d20e803d356627e208deb6fe2 2023-03-20 10:57:45,279 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=72402f7d20e803d356627e208deb6fe2, regionState=CLOSED 2023-03-20 10:57:45,279 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679309865278"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679309865278"}]},"ts":"1679309865278"} 2023-03-20 10:57:45,281 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=32 2023-03-20 10:57:45,281 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=32, state=SUCCESS; CloseRegionProcedure 72402f7d20e803d356627e208deb6fe2, server=jenkins-hbase20.apache.org,36041,1679309777062 in 1.1620 sec 2023-03-20 10:57:45,281 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=31 2023-03-20 10:57:45,282 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=31, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=72402f7d20e803d356627e208deb6fe2, UNASSIGN in 1.1670 sec 2023-03-20 10:57:45,282 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679309865282"}]},"ts":"1679309865282"} 2023-03-20 10:57:45,283 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-20 10:57:45,284 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-20 10:57:45,285 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=31, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.2730 sec 2023-03-20 10:57:46,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-20 10:57:46,122 INFO [Listener at localhost.localdomain/43509] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 31 completed 2023-03-20 10:57:46,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-03-20 10:57:46,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure2.ProcedureExecutor(1029): Stored pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-20 10:57:46,125 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-20 10:57:46,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-03-20 10:57:46,125 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=34, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-20 10:57:46,129 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2 2023-03-20 10:57:46,131 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A, FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B, FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C, FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/recovered.edits] 2023-03-20 10:57:46,134 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/1c72e53945b641a287b3d4d537a735bf to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/1c72e53945b641a287b3d4d537a735bf 2023-03-20 10:57:46,135 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/31c06de0cb36493c88a1ab5ed833e72d to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/A/31c06de0cb36493c88a1ab5ed833e72d 2023-03-20 10:57:46,137 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/4253299e165b479ea4649df65f0ce1b1 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/4253299e165b479ea4649df65f0ce1b1 2023-03-20 10:57:46,138 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/5fc6ef2e56d74858945605198b5c770b to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/B/5fc6ef2e56d74858945605198b5c770b 2023-03-20 10:57:46,140 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/d6c2f00470d848f8a88825ddff07c238 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/d6c2f00470d848f8a88825ddff07c238 2023-03-20 10:57:46,141 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/d99204e6541c4d2ca14277dd0c9032e9 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/C/d99204e6541c4d2ca14277dd0c9032e9 2023-03-20 10:57:46,143 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/recovered.edits/1345.seqid to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2/recovered.edits/1345.seqid 2023-03-20 10:57:46,143 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/72402f7d20e803d356627e208deb6fe2 2023-03-20 10:57:46,144 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-20 10:57:46,145 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=34, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-20 10:57:46,146 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-20 10:57:46,147 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-20 10:57:46,148 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=34, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-20 10:57:46,148 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-20 10:57:46,148 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1679309866148"}]},"ts":"9223372036854775807"} 2023-03-20 10:57:46,149 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-20 10:57:46,150 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 72402f7d20e803d356627e208deb6fe2, NAME => 'TestAcidGuarantees,,1679309838628.72402f7d20e803d356627e208deb6fe2.', STARTKEY => '', ENDKEY => ''}] 2023-03-20 10:57:46,150 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-20 10:57:46,150 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1679309866150"}]},"ts":"9223372036854775807"} 2023-03-20 10:57:46,151 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-20 10:57:46,152 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=34, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-20 10:57:46,153 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=34, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 29 msec 2023-03-20 10:57:46,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-03-20 10:57:46,227 INFO [Listener at localhost.localdomain/43509] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 34 completed 2023-03-20 10:57:46,237 INFO [Listener at localhost.localdomain/43509] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=228 (was 230), OpenFileDescriptor=555 (was 545) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=842 (was 782) - SystemLoadAverage LEAK? -, ProcessCount=184 (was 184), AvailableMemoryMB=8434 (was 8771) 2023-03-20 10:57:46,250 INFO [Listener at localhost.localdomain/43509] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=228, OpenFileDescriptor=555, MaxFileDescriptor=60000, SystemLoadAverage=842, ProcessCount=184, AvailableMemoryMB=8434 2023-03-20 10:57:46,252 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-20 10:57:46,252 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-20 10:57:46,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-20 10:57:46,255 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-20 10:57:46,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 35 2023-03-20 10:57:46,255 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-20 10:57:46,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-20 10:57:46,256 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991 2023-03-20 10:57:46,256 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991 empty. 2023-03-20 10:57:46,257 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991 2023-03-20 10:57:46,257 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-20 10:57:46,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-20 10:57:46,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-20 10:57:46,663 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-20 10:57:46,665 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => ee10ffd90d917f64b63db6b593dd0991, NAME => 'TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp 2023-03-20 10:57:46,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-20 10:57:47,071 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-20 10:57:47,071 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing ee10ffd90d917f64b63db6b593dd0991, disabling compactions & flushes 2023-03-20 10:57:47,071 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:47,071 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:47,071 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. after waiting 0 ms 2023-03-20 10:57:47,071 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:47,071 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:47,071 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for ee10ffd90d917f64b63db6b593dd0991: 2023-03-20 10:57:47,073 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-20 10:57:47,074 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679309867074"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679309867074"}]},"ts":"1679309867074"} 2023-03-20 10:57:47,075 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-20 10:57:47,076 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-20 10:57:47,076 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679309867076"}]},"ts":"1679309867076"} 2023-03-20 10:57:47,077 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-20 10:57:47,080 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ee10ffd90d917f64b63db6b593dd0991, ASSIGN}] 2023-03-20 10:57:47,081 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ee10ffd90d917f64b63db6b593dd0991, ASSIGN 2023-03-20 10:57:47,082 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=ee10ffd90d917f64b63db6b593dd0991, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,36041,1679309777062; forceNewPlan=false, retain=false 2023-03-20 10:57:47,233 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=ee10ffd90d917f64b63db6b593dd0991, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:47,233 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679309867233"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679309867233"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679309867233"}]},"ts":"1679309867233"} 2023-03-20 10:57:47,235 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE; OpenRegionProcedure ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062}] 2023-03-20 10:57:47,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-20 10:57:47,391 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:47,391 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => ee10ffd90d917f64b63db6b593dd0991, NAME => 'TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.', STARTKEY => '', ENDKEY => ''} 2023-03-20 10:57:47,392 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees ee10ffd90d917f64b63db6b593dd0991 2023-03-20 10:57:47,392 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-20 10:57:47,392 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for ee10ffd90d917f64b63db6b593dd0991 2023-03-20 10:57:47,392 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for ee10ffd90d917f64b63db6b593dd0991 2023-03-20 10:57:47,398 INFO [StoreOpener-ee10ffd90d917f64b63db6b593dd0991-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region ee10ffd90d917f64b63db6b593dd0991 2023-03-20 10:57:47,404 DEBUG [StoreOpener-ee10ffd90d917f64b63db6b593dd0991-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A 2023-03-20 10:57:47,404 DEBUG [StoreOpener-ee10ffd90d917f64b63db6b593dd0991-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A 2023-03-20 10:57:47,404 INFO [StoreOpener-ee10ffd90d917f64b63db6b593dd0991-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-20 10:57:47,404 INFO [StoreOpener-ee10ffd90d917f64b63db6b593dd0991-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ee10ffd90d917f64b63db6b593dd0991 columnFamilyName A 2023-03-20 10:57:47,405 INFO [StoreOpener-ee10ffd90d917f64b63db6b593dd0991-1] regionserver.HStore(310): Store=ee10ffd90d917f64b63db6b593dd0991/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-20 10:57:47,405 INFO [StoreOpener-ee10ffd90d917f64b63db6b593dd0991-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region ee10ffd90d917f64b63db6b593dd0991 2023-03-20 10:57:47,410 DEBUG [StoreOpener-ee10ffd90d917f64b63db6b593dd0991-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B 2023-03-20 10:57:47,410 DEBUG [StoreOpener-ee10ffd90d917f64b63db6b593dd0991-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B 2023-03-20 10:57:47,410 INFO [StoreOpener-ee10ffd90d917f64b63db6b593dd0991-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-20 10:57:47,410 INFO [StoreOpener-ee10ffd90d917f64b63db6b593dd0991-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ee10ffd90d917f64b63db6b593dd0991 columnFamilyName B 2023-03-20 10:57:47,411 INFO [StoreOpener-ee10ffd90d917f64b63db6b593dd0991-1] regionserver.HStore(310): Store=ee10ffd90d917f64b63db6b593dd0991/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-20 10:57:47,412 INFO [StoreOpener-ee10ffd90d917f64b63db6b593dd0991-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region ee10ffd90d917f64b63db6b593dd0991 2023-03-20 10:57:47,413 DEBUG [StoreOpener-ee10ffd90d917f64b63db6b593dd0991-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C 2023-03-20 10:57:47,413 DEBUG [StoreOpener-ee10ffd90d917f64b63db6b593dd0991-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C 2023-03-20 10:57:47,413 INFO [StoreOpener-ee10ffd90d917f64b63db6b593dd0991-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-20 10:57:47,414 INFO [StoreOpener-ee10ffd90d917f64b63db6b593dd0991-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ee10ffd90d917f64b63db6b593dd0991 columnFamilyName C 2023-03-20 10:57:47,415 INFO [StoreOpener-ee10ffd90d917f64b63db6b593dd0991-1] regionserver.HStore(310): Store=ee10ffd90d917f64b63db6b593dd0991/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-20 10:57:47,416 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:47,416 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991 2023-03-20 10:57:47,417 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991 2023-03-20 10:57:47,419 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-20 10:57:47,420 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for ee10ffd90d917f64b63db6b593dd0991 2023-03-20 10:57:47,424 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-20 10:57:47,424 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened ee10ffd90d917f64b63db6b593dd0991; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73873481, jitterRate=0.10080064833164215}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-20 10:57:47,424 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for ee10ffd90d917f64b63db6b593dd0991: 2023-03-20 10:57:47,425 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991., pid=37, masterSystemTime=1679309867386 2023-03-20 10:57:47,429 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:47,429 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:47,430 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=ee10ffd90d917f64b63db6b593dd0991, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:47,430 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679309867430"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1679309867430"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679309867430"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679309867430"}]},"ts":"1679309867430"} 2023-03-20 10:57:47,433 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-03-20 10:57:47,433 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; OpenRegionProcedure ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 in 196 msec 2023-03-20 10:57:47,435 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=36, resume processing ppid=35 2023-03-20 10:57:47,435 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=ee10ffd90d917f64b63db6b593dd0991, ASSIGN in 353 msec 2023-03-20 10:57:47,436 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-20 10:57:47,436 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679309867436"}]},"ts":"1679309867436"} 2023-03-20 10:57:47,437 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-20 10:57:47,439 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-20 10:57:47,441 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1870 sec 2023-03-20 10:57:48,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-20 10:57:48,360 INFO [Listener at localhost.localdomain/43509] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 35 completed 2023-03-20 10:57:48,361 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(139): Connect 0x7d243e46 to 127.0.0.1:60226 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 10:57:48,366 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@29738198, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 10:57:48,367 DEBUG [Listener at localhost.localdomain/43509] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 10:57:48,372 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43468, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 10:57:48,373 DEBUG [Listener at localhost.localdomain/43509] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-20 10:57:48,374 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:35306, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-20 10:57:48,375 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-20 10:57:48,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.HMaster$12(2454): Client=jenkins//148.251.75.209 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-20 10:57:48,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure2.ProcedureExecutor(1029): Stored pid=38, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-03-20 10:57:48,788 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-20 10:57:48,789 DEBUG [PEWorker-2] util.FSTableDescriptors(519): Deleted hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-20 10:57:48,789 INFO [PEWorker-2] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-20 10:57:48,791 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=38, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-03-20 10:57:48,794 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ee10ffd90d917f64b63db6b593dd0991, REOPEN/MOVE}] 2023-03-20 10:57:48,794 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ee10ffd90d917f64b63db6b593dd0991, REOPEN/MOVE 2023-03-20 10:57:48,795 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=ee10ffd90d917f64b63db6b593dd0991, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:48,795 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679309868795"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679309868795"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679309868795"}]},"ts":"1679309868795"} 2023-03-20 10:57:48,796 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=41, ppid=40, state=RUNNABLE; CloseRegionProcedure ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062}] 2023-03-20 10:57:48,948 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close ee10ffd90d917f64b63db6b593dd0991 2023-03-20 10:57:48,949 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing ee10ffd90d917f64b63db6b593dd0991, disabling compactions & flushes 2023-03-20 10:57:48,949 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:48,949 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:48,949 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. after waiting 0 ms 2023-03-20 10:57:48,949 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:48,952 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-03-20 10:57:48,953 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:48,953 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for ee10ffd90d917f64b63db6b593dd0991: 2023-03-20 10:57:48,953 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(3509): Not adding moved region record: ee10ffd90d917f64b63db6b593dd0991 to self. 2023-03-20 10:57:48,954 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed ee10ffd90d917f64b63db6b593dd0991 2023-03-20 10:57:48,954 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=ee10ffd90d917f64b63db6b593dd0991, regionState=CLOSED 2023-03-20 10:57:48,954 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679309868954"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679309868954"}]},"ts":"1679309868954"} 2023-03-20 10:57:48,956 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=40 2023-03-20 10:57:48,957 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=40, state=SUCCESS; CloseRegionProcedure ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 in 159 msec 2023-03-20 10:57:48,957 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=ee10ffd90d917f64b63db6b593dd0991, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase20.apache.org,36041,1679309777062; forceNewPlan=false, retain=true 2023-03-20 10:57:49,107 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=ee10ffd90d917f64b63db6b593dd0991, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,108 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679309869107"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679309869107"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679309869107"}]},"ts":"1679309869107"} 2023-03-20 10:57:49,109 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=42, ppid=40, state=RUNNABLE; OpenRegionProcedure ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062}] 2023-03-20 10:57:49,264 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:49,264 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => ee10ffd90d917f64b63db6b593dd0991, NAME => 'TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.', STARTKEY => '', ENDKEY => ''} 2023-03-20 10:57:49,265 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees ee10ffd90d917f64b63db6b593dd0991 2023-03-20 10:57:49,265 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-20 10:57:49,265 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for ee10ffd90d917f64b63db6b593dd0991 2023-03-20 10:57:49,265 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for ee10ffd90d917f64b63db6b593dd0991 2023-03-20 10:57:49,266 INFO [StoreOpener-ee10ffd90d917f64b63db6b593dd0991-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region ee10ffd90d917f64b63db6b593dd0991 2023-03-20 10:57:49,267 DEBUG [StoreOpener-ee10ffd90d917f64b63db6b593dd0991-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A 2023-03-20 10:57:49,267 DEBUG [StoreOpener-ee10ffd90d917f64b63db6b593dd0991-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A 2023-03-20 10:57:49,267 INFO [StoreOpener-ee10ffd90d917f64b63db6b593dd0991-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-20 10:57:49,267 INFO [StoreOpener-ee10ffd90d917f64b63db6b593dd0991-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ee10ffd90d917f64b63db6b593dd0991 columnFamilyName A 2023-03-20 10:57:49,269 INFO [StoreOpener-ee10ffd90d917f64b63db6b593dd0991-1] regionserver.HStore(310): Store=ee10ffd90d917f64b63db6b593dd0991/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-20 10:57:49,269 INFO [StoreOpener-ee10ffd90d917f64b63db6b593dd0991-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region ee10ffd90d917f64b63db6b593dd0991 2023-03-20 10:57:49,270 DEBUG [StoreOpener-ee10ffd90d917f64b63db6b593dd0991-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B 2023-03-20 10:57:49,270 DEBUG [StoreOpener-ee10ffd90d917f64b63db6b593dd0991-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B 2023-03-20 10:57:49,270 INFO [StoreOpener-ee10ffd90d917f64b63db6b593dd0991-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-20 10:57:49,271 INFO [StoreOpener-ee10ffd90d917f64b63db6b593dd0991-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ee10ffd90d917f64b63db6b593dd0991 columnFamilyName B 2023-03-20 10:57:49,271 INFO [StoreOpener-ee10ffd90d917f64b63db6b593dd0991-1] regionserver.HStore(310): Store=ee10ffd90d917f64b63db6b593dd0991/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-20 10:57:49,271 INFO [StoreOpener-ee10ffd90d917f64b63db6b593dd0991-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region ee10ffd90d917f64b63db6b593dd0991 2023-03-20 10:57:49,272 DEBUG [StoreOpener-ee10ffd90d917f64b63db6b593dd0991-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C 2023-03-20 10:57:49,272 DEBUG [StoreOpener-ee10ffd90d917f64b63db6b593dd0991-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C 2023-03-20 10:57:49,272 INFO [StoreOpener-ee10ffd90d917f64b63db6b593dd0991-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-20 10:57:49,272 INFO [StoreOpener-ee10ffd90d917f64b63db6b593dd0991-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ee10ffd90d917f64b63db6b593dd0991 columnFamilyName C 2023-03-20 10:57:49,273 INFO [StoreOpener-ee10ffd90d917f64b63db6b593dd0991-1] regionserver.HStore(310): Store=ee10ffd90d917f64b63db6b593dd0991/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-20 10:57:49,273 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:49,274 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991 2023-03-20 10:57:49,275 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991 2023-03-20 10:57:49,277 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-20 10:57:49,278 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for ee10ffd90d917f64b63db6b593dd0991 2023-03-20 10:57:49,279 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened ee10ffd90d917f64b63db6b593dd0991; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68472379, jitterRate=0.020317956805229187}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-20 10:57:49,279 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for ee10ffd90d917f64b63db6b593dd0991: 2023-03-20 10:57:49,279 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991., pid=42, masterSystemTime=1679309869260 2023-03-20 10:57:49,280 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:49,281 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:49,281 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=ee10ffd90d917f64b63db6b593dd0991, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,281 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679309869281"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1679309869281"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679309869281"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679309869281"}]},"ts":"1679309869281"} 2023-03-20 10:57:49,282 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=40 2023-03-20 10:57:49,283 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=40, state=SUCCESS; OpenRegionProcedure ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 in 173 msec 2023-03-20 10:57:49,283 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=40, resume processing ppid=39 2023-03-20 10:57:49,284 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=39, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=ee10ffd90d917f64b63db6b593dd0991, REOPEN/MOVE in 488 msec 2023-03-20 10:57:49,285 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=38 2023-03-20 10:57:49,285 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=38, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 493 msec 2023-03-20 10:57:49,286 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=38, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 909 msec 2023-03-20 10:57:49,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=38 2023-03-20 10:57:49,288 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(139): Connect 0x74a355de to 127.0.0.1:60226 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 10:57:49,292 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@28a98302, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 10:57:49,293 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(139): Connect 0x57b6628e to 127.0.0.1:60226 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 10:57:49,304 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@39d5c003, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 10:57:49,304 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(139): Connect 0x5da5030e to 127.0.0.1:60226 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 10:57:49,308 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@39e3a826, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 10:57:49,308 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(139): Connect 0x74d6bf18 to 127.0.0.1:60226 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 10:57:49,312 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@19e7fff, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 10:57:49,312 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(139): Connect 0x060c5cba to 127.0.0.1:60226 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 10:57:49,315 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@47e30d1c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 10:57:49,316 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(139): Connect 0x36a9ece3 to 127.0.0.1:60226 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 10:57:49,319 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@69fd9cc7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 10:57:49,320 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(139): Connect 0x2ded66ad to 127.0.0.1:60226 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 10:57:49,324 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@54da4907, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 10:57:49,325 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(139): Connect 0x14d061d2 to 127.0.0.1:60226 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 10:57:49,330 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@52cea5eb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 10:57:49,330 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(139): Connect 0x228ff1a4 to 127.0.0.1:60226 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 10:57:49,334 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3546dad6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 10:57:49,334 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(139): Connect 0x143a1290 to 127.0.0.1:60226 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 10:57:49,339 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4965c362, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 10:57:49,342 DEBUG [hconnection-0x3a0d53d0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 10:57:49,344 DEBUG [hconnection-0x64d4776a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 10:57:49,344 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43478, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 10:57:49,351 DEBUG [hconnection-0x1de971f4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 10:57:49,351 DEBUG [hconnection-0x31cff851-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 10:57:49,351 DEBUG [hconnection-0x533e8a37-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 10:57:49,351 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43484, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 10:57:49,352 DEBUG [hconnection-0x15482e06-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 10:57:49,352 DEBUG [hconnection-0x6951dd76-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 10:57:49,353 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43492, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 10:57:49,353 DEBUG [hconnection-0x1ac4174b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 10:57:49,353 DEBUG [hconnection-0x6ce9a49d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 10:57:49,355 DEBUG [hconnection-0x1054a9de-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 10:57:49,357 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43528, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 10:57:49,359 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43568, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 10:57:49,361 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43512, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 10:57:49,362 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43544, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 10:57:49,364 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43494, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 10:57:49,366 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43530, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 10:57:49,366 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:43552, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 10:57:49,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:49,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:49,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:49,375 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:49,375 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:49,375 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:49,375 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:49,375 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ee10ffd90d917f64b63db6b593dd0991 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-20 10:57:49,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on ee10ffd90d917f64b63db6b593dd0991 2023-03-20 10:57:49,375 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:49,376 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:49,376 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee10ffd90d917f64b63db6b593dd0991, store=A 2023-03-20 10:57:49,376 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:49,376 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee10ffd90d917f64b63db6b593dd0991, store=B 2023-03-20 10:57:49,376 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:49,376 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee10ffd90d917f64b63db6b593dd0991, store=C 2023-03-20 10:57:49,376 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:49,377 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,377 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,377 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:49,379 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:49,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:49,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:49,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:49,380 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:49,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:49,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:49,380 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:49,385 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:49,385 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:49,386 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:49,386 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:49,386 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:49,387 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:49,387 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:49,392 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:49,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 5 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43494 deadline: 1679309929390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,392 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:49,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43530 deadline: 1679309929391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,393 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:49,393 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:49,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43492 deadline: 1679309929392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43528 deadline: 1679309929392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,393 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1679309869384, max timestamp=1679309869392Num uniques 859; 2023-03-20 10:57:49,394 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1679309869384, max timestamp=1679309869392Num uniques 859; 2023-03-20 10:57:49,394 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-03-20 10:57:49,394 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-03-20 10:57:49,394 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:49,394 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1679309869384, max timestamp=1679309869392Num uniques 859; 2023-03-20 10:57:49,394 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. started... 2023-03-20 10:57:49,394 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. as already flushing 2023-03-20 10:57:49,394 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-03-20 10:57:49,394 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:49,394 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:49,394 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:49,394 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,394 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:49,394 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:49,395 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:49,395 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:49,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:49,395 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:49,395 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:49,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:49,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:49,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:49,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:49,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:49,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@80c143[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:49,403 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:49,403 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:49,404 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:49,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:49,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:49,404 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:49,404 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:49,404 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:49,405 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,405 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:49,405 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:49,405 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:49,405 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:49,405 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:49,405 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:49,405 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:49,406 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:49,406 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:49,406 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:49,407 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:49,407 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:49,407 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:49,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:49,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:49,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:49,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:49,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:49,407 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:49,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:49,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:49,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:49,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:49,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:49,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:49,408 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:49,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:49,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,410 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:49,410 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:49,410 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,410 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:49,410 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:49,410 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,410 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:49,410 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:49,410 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:49,410 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:49,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:49,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:49,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:49,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:49,411 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:57:49,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:49,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:49,411 DEBUG [Thread-2246] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:49,411 DEBUG [Thread-2246] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:49,424 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:49,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43478 deadline: 1679309929420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,499 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:49,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43492 deadline: 1679309929494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,499 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:49,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43528 deadline: 1679309929494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,500 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:49,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43530 deadline: 1679309929495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,500 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:49,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43494 deadline: 1679309929496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,511 DEBUG [Thread-2246] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:49,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:49,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:49,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:49,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:49,518 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:49,518 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:49,518 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:49,518 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:49,518 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:49,518 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:49,525 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:49,525 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:49,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:49,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:49,526 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,526 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:49,526 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:49,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:49,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:49,527 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:49,530 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:49,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43478 deadline: 1679309929527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,531 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:49,531 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:49,531 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:49,531 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:49,531 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:49,538 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:49,538 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:49,539 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:49,539 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. started... 2023-03-20 10:57:49,539 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. as already flushing 2023-03-20 10:57:49,539 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:49,539 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:49,539 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:49,539 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,539 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:49,539 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:49,540 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,540 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:49,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:49,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:49,540 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:49,540 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:49,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:49,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:49,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:49,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:49,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:49,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62b8358e[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:49,542 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:49,542 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:49,543 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:49,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:49,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:49,543 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:49,543 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:49,543 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:49,543 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,543 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:49,544 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:49,544 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,544 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:49,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:49,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:49,544 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:49,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:49,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:49,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:49,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:49,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:49,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:49,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:49,547 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:49,547 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:49,547 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:49,547 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:49,547 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:49,548 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:49,548 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:49,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:49,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:49,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:49,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:49,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:49,548 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:49,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:49,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:49,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:49,549 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:49,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:49,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:49,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:49,549 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:49,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:49,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,551 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,551 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:49,551 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:49,551 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:49,551 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:49,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:49,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:49,551 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,551 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:49,551 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:49,551 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:49,551 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:57:49,551 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:49,552 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:49,551 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:49,552 DEBUG [Thread-2246] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:49,552 DEBUG [Thread-2246] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:49,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:49,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:49,652 DEBUG [Thread-2246] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:49,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:49,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:49,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:49,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:49,670 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:49,670 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:49,670 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:49,670 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:49,670 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:49,670 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:49,671 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:49,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:49,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:49,671 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,671 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,671 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:49,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:49,672 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:49,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:49,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:49,672 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:49,674 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:49,674 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:49,674 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:49,674 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:49,674 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:49,683 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:49,683 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:49,683 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:49,683 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. started... 2023-03-20 10:57:49,683 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. as already flushing 2023-03-20 10:57:49,683 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:49,684 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:49,684 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:49,684 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,684 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:49,684 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:49,687 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,687 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:49,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:49,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:49,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:49,687 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:49,687 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:49,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:49,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:49,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:49,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:49,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58ce7b88[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:49,688 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:49,688 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:49,689 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,689 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:49,690 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:49,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:49,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:49,690 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:49,690 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:49,690 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:49,692 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:49,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:49,693 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:49,693 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:49,693 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:49,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:49,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:49,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:49,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:49,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:49,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:49,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:49,696 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:49,696 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:49,696 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:49,696 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:49,696 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:49,698 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:49,698 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:49,698 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:49,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:49,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:49,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:49,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:49,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:49,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:49,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:49,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:49,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:49,699 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:49,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:49,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:49,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,701 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,702 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:49,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43492 deadline: 1679309929700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,702 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:49,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43530 deadline: 1679309929701, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,702 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:49,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43528 deadline: 1679309929701, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,703 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:49,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43494 deadline: 1679309929703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:49,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:49,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,711 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,711 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:49,711 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:49,712 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:49,712 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,712 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:49,712 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:49,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:49,712 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:57:49,712 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:49,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:49,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:49,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:49,712 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:49,712 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:49,712 DEBUG [Thread-2246] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:49,712 DEBUG [Thread-2246] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:49,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:49,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:49,738 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:49,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43478 deadline: 1679309929731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,816 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303203594524ae8ee458db76da43d2100c840 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303203594524ae8ee458db76da43d2100c840 2023-03-20 10:57:49,818 DEBUG [Thread-2246] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:49,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:49,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:49,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:49,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:49,855 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:49,855 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:49,855 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:49,855 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:49,856 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:49,856 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:49,856 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,856 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:49,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:49,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:49,857 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,857 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:49,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:49,857 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:49,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:49,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:49,858 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:49,867 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:49,867 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:49,867 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:49,867 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:49,867 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:49,879 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:49,879 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:49,879 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:49,879 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. started... 2023-03-20 10:57:49,879 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. as already flushing 2023-03-20 10:57:49,879 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:49,879 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:49,879 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:49,879 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,880 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:49,880 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:49,881 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:49,881 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:49,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:49,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:49,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:49,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:49,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:49,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:49,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54b2c121[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:49,885 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:49,885 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:49,886 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,886 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:49,887 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:49,888 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:49,888 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:49,888 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:49,888 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:49,888 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:49,890 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:49,890 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:49,890 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:49,890 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:49,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:49,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:49,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:49,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:49,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:49,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:49,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:49,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:49,908 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:49,908 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:49,908 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:49,908 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:49,908 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:49,910 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:49,910 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:49,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:49,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:49,910 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:49,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:49,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:49,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:49,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:49,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:49,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:49,911 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:49,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:49,912 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:49,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:49,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:49,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:49,914 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,914 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:49,914 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:49,915 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:49,915 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:49,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:49,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:49,915 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:49,915 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:49,915 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:49,915 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:49,915 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:49,915 DEBUG [Thread-2246] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:49,915 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:49,915 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:57:49,915 DEBUG [Thread-2246] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:49,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:49,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:49,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:49,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:50,014 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:50,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43528 deadline: 1679309930005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,014 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:50,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43492 deadline: 1679309930005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,015 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:50,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43494 deadline: 1679309930007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,015 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:50,015 DEBUG [Thread-2246] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:50,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43530 deadline: 1679309930007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:50,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:50,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:50,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:50,026 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:50,026 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:50,026 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:50,026 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:50,027 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:50,027 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:50,027 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,028 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,028 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:50,028 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:50,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:50,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:50,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:50,031 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:50,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:50,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:50,032 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:50,042 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:50,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43478 deadline: 1679309930041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,046 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:50,046 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:50,046 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:50,046 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:50,046 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:50,050 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:50,050 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:50,054 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:50,054 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. started... 2023-03-20 10:57:50,054 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. as already flushing 2023-03-20 10:57:50,055 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:50,055 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:50,055 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:50,055 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,055 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:50,055 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:50,056 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:50,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:50,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:50,057 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:50,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:50,057 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:50,057 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:50,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:50,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:50,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:50,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52be882d[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:50,058 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:50,058 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:50,059 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:50,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:50,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:50,059 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:50,059 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:50,059 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:50,060 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,060 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:50,061 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:50,061 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:50,061 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:50,061 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:50,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:50,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:50,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:50,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:50,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:50,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:50,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:50,064 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:50,065 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:50,065 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:50,065 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:50,065 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:50,065 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:50,070 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:50,070 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:50,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:50,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:50,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:50,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:50,070 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:50,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:50,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:50,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:50,071 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:50,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:50,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:50,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:50,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:50,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,073 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:50,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:50,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,078 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:50,078 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:50,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:50,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:50,078 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,079 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:50,079 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:50,079 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:50,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:50,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:50,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:50,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:50,079 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,082 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:50,082 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:50,082 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:50,082 DEBUG [Thread-2246] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:50,082 DEBUG [Thread-2246] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:50,182 DEBUG [Thread-2246] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:50,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:50,184 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:50,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:50,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:50,197 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:50,197 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:50,197 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:50,197 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:50,197 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:50,198 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:50,198 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:50,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:50,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:50,198 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,199 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,199 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:50,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:50,199 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:50,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:50,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:50,200 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:50,200 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:50,200 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:50,200 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:50,200 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:50,200 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:50,206 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:50,206 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:50,206 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:50,206 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. started... 2023-03-20 10:57:50,206 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. as already flushing 2023-03-20 10:57:50,206 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:50,206 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:50,206 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:50,206 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,206 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:50,206 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:50,212 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:50,212 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:50,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:50,212 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:50,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:50,212 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:50,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:50,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:50,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:50,213 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:50,213 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:50,213 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7109d3b1[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:50,214 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:50,214 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:50,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:50,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:50,215 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:50,215 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:50,215 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:50,216 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:50,216 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:50,216 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,216 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:50,217 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:50,217 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:50,217 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:50,217 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:50,217 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:50,217 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:50,217 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:50,218 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:50,218 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:50,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:50,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:50,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:50,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:50,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:50,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:50,218 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:50,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:50,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:50,218 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:50,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:50,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:50,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:50,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,219 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:50,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:50,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,222 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:50,222 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:50,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:50,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:50,224 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,224 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:50,224 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:50,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:50,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:50,224 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,224 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:50,224 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:50,224 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:50,224 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:50,224 DEBUG [Thread-2246] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:50,225 DEBUG [Thread-2246] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:50,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:50,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:50,248 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=16, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/A/0602a43a94cd4cdf880607998644fb04 2023-03-20 10:57:50,325 DEBUG [Thread-2246] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:50,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:50,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:50,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:50,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:50,342 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:50,342 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:50,342 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:50,342 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:50,343 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:50,343 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:50,343 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:50,343 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:50,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:50,344 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/B/8167b7abdc364b828f2f5a16e552b1b5 2023-03-20 10:57:50,344 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,344 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:50,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:50,344 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:50,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:50,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:50,345 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:50,362 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:50,362 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:50,362 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:50,362 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:50,381 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:50,381 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:50,381 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:50,381 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. started... 2023-03-20 10:57:50,381 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. as already flushing 2023-03-20 10:57:50,381 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:50,381 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:50,381 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:50,381 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,381 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:50,381 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:50,383 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:50,383 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:50,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:50,383 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:50,383 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:50,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:50,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:50,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:50,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:50,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:50,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a30e423[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:50,386 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:50,386 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:50,387 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:50,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:50,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:50,387 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,387 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:50,387 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:50,387 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:50,387 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:50,390 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,390 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:50,390 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:50,390 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:50,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:50,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:50,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:50,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:50,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:50,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:50,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:50,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:50,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:50,392 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:50,392 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:50,392 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:50,392 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:50,392 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:50,401 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/C/00f4ffb13fb54e29b53d38a691eadd1c 2023-03-20 10:57:50,403 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:50,403 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:50,404 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:50,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:50,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:50,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:50,405 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/A/0602a43a94cd4cdf880607998644fb04 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/0602a43a94cd4cdf880607998644fb04 2023-03-20 10:57:50,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:50,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:50,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:50,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:50,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:50,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:50,408 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:50,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:50,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:50,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,410 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:50,411 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/0602a43a94cd4cdf880607998644fb04, entries=200, sequenceid=16, filesize=31.9 K 2023-03-20 10:57:50,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:50,411 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/B/8167b7abdc364b828f2f5a16e552b1b5 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/8167b7abdc364b828f2f5a16e552b1b5 2023-03-20 10:57:50,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,415 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:50,415 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:50,415 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,415 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:50,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:50,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:50,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:50,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:50,415 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:50,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:50,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:50,416 DEBUG [Thread-2246] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:50,416 DEBUG [Thread-2246] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:50,415 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:50,416 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,416 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:50,416 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:50,416 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:50,417 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/8167b7abdc364b828f2f5a16e552b1b5, entries=150, sequenceid=16, filesize=11.6 K 2023-03-20 10:57:50,417 DEBUG [MemStoreFlusher.0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-03-20 10:57:50,418 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/C/00f4ffb13fb54e29b53d38a691eadd1c as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/00f4ffb13fb54e29b53d38a691eadd1c 2023-03-20 10:57:50,429 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/00f4ffb13fb54e29b53d38a691eadd1c, entries=150, sequenceid=16, filesize=11.6 K 2023-03-20 10:57:50,431 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=127.47 KB/130530 for ee10ffd90d917f64b63db6b593dd0991 in 1056ms, sequenceid=16, compaction requested=false 2023-03-20 10:57:50,431 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-03-20 10:57:50,432 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ee10ffd90d917f64b63db6b593dd0991: 2023-03-20 10:57:50,516 DEBUG [Thread-2246] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:50,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:50,520 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:50,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:50,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:50,521 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:50,521 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:50,521 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:50,521 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:50,521 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:50,521 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:50,523 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:50,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:50,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:50,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:50,523 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on ee10ffd90d917f64b63db6b593dd0991 2023-03-20 10:57:50,524 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ee10ffd90d917f64b63db6b593dd0991 3/3 column families, dataSize=134.18 KB heapSize=361.55 KB 2023-03-20 10:57:50,524 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee10ffd90d917f64b63db6b593dd0991, store=A 2023-03-20 10:57:50,524 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:57:50,524 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee10ffd90d917f64b63db6b593dd0991, store=B 2023-03-20 10:57:50,524 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:57:50,524 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee10ffd90d917f64b63db6b593dd0991, store=C 2023-03-20 10:57:50,524 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:57:50,524 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,524 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:50,524 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:50,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:50,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:50,525 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:50,526 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:50,526 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:50,526 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:50,526 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:50,526 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:50,528 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:50,528 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:50,528 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:50,528 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. started... 2023-03-20 10:57:50,528 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. as already flushing 2023-03-20 10:57:50,528 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:50,528 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:50,528 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:50,528 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,528 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:50,528 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:50,536 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,536 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:50,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:50,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:50,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:50,536 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:50,536 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:50,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:50,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:50,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:50,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:50,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18c98d8d[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:50,537 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:50,538 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:50,538 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:50,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:50,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:50,538 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:50,538 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:50,538 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:50,539 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,539 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:50,539 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,539 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:50,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:50,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:50,539 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:50,540 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:50,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:50,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:50,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:50,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:50,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:50,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:50,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:50,542 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:50,542 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:50,542 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:50,542 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:50,542 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:50,543 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:50,543 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:50,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:50,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:50,543 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:50,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:50,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:50,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:50,545 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:50,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43494 deadline: 1679309930535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,545 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:50,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43530 deadline: 1679309930535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:50,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:50,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:50,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:50,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:50,548 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:50,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:50,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:50,549 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:50,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,553 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,553 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:50,553 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:50,553 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,553 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:50,553 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:50,553 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:50,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:50,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:50,553 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:50,553 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:50,553 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:50,553 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:50,554 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:50,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:50,554 DEBUG [Thread-2246] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:50,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:50,554 DEBUG [Thread-2246] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:50,555 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:50,555 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:50,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43492 deadline: 1679309930546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43528 deadline: 1679309930546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,555 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:50,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43478 deadline: 1679309930550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,654 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:50,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43530 deadline: 1679309930647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,654 DEBUG [Thread-2246] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:50,654 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:50,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43494 deadline: 1679309930652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:50,659 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:50,662 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:50,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43492 deadline: 1679309930658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,663 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:50,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43528 deadline: 1679309930660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:50,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:50,666 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:50,666 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:50,666 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:50,666 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:50,667 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:50,667 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:50,667 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,667 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:50,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:50,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:50,668 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,668 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:50,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:50,670 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:50,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:50,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:50,670 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:50,671 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:50,671 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:50,671 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:50,671 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:50,671 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:50,671 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:50,671 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:50,671 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:50,671 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. started... 2023-03-20 10:57:50,671 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. as already flushing 2023-03-20 10:57:50,671 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:50,671 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:50,671 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:50,671 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,671 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:50,671 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:50,678 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:50,678 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:50,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:50,678 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:50,678 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:50,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:50,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:50,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:50,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:50,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:50,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e908159[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:50,680 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:50,681 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:50,687 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,687 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:50,689 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:50,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:50,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:50,689 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:50,689 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:50,689 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:50,695 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,695 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:50,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:50,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:50,695 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:50,695 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:50,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:50,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:50,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:50,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:50,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:50,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:50,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:50,697 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:50,697 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:50,697 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:50,697 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:50,697 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:50,703 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:50,703 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:50,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:50,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:50,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:50,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:50,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:50,703 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:50,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:50,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:50,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:50,704 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:50,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:50,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:50,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:50,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,705 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:50,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:50,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,713 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:50,713 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,713 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:50,713 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:50,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:50,713 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:57:50,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:50,713 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:50,713 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,713 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:50,713 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:50,713 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:50,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:50,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:50,714 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:50,717 DEBUG [Thread-2246] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:50,717 DEBUG [Thread-2246] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:50,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:50,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:50,817 DEBUG [Thread-2246] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:50,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:50,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:50,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:50,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:50,834 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:50,834 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:50,834 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:50,834 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:50,840 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:50,840 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:50,844 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,844 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,844 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:50,846 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:50,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:50,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:50,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:50,850 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:50,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:50,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:50,853 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:50,853 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-20 10:57:50,853 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:50,853 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:50,853 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:50,853 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:50,853 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:50,856 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:50,856 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:50,858 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:50,858 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. started... 2023-03-20 10:57:50,858 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. as already flushing 2023-03-20 10:57:50,858 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:50,858 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:50,858 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:50,858 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,860 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:50,860 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:50,867 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:50,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43530 deadline: 1679309930857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,867 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:50,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43494 deadline: 1679309930857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,872 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:50,872 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:50,872 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:50,874 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:50,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:50,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:50,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:50,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:50,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:50,879 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:50,879 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1bb4961[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:50,879 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:50,879 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:50,880 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,880 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:50,882 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:50,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:50,882 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:50,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43528 deadline: 1679309930867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:50,882 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:50,882 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:50,882 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:50,883 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:50,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43492 deadline: 1679309930867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,884 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:50,884 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:50,884 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:50,886 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:50,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:50,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:50,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:50,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:50,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:50,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:50,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:50,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:50,890 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:50,890 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:50,890 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:50,891 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:50,891 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:50,896 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:50,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:50,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:50,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:50,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:50,898 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,898 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:50,898 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:50,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:50,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:50,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:50,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:50,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:50,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,900 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:50,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:50,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,905 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:50,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:50,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:50,905 DEBUG [Thread-2246] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:50,905 DEBUG [Thread-2246] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:50,908 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,908 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:50,908 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:50,908 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:50,908 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:50,908 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:50,908 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:50,908 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:57:50,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:50,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:50,910 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:50,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:50,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:50,950 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230320574f61896b4b495face029905a816368 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320574f61896b4b495face029905a816368 2023-03-20 10:57:50,994 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=48, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/A/1decfdfeb5f545388e5e811c1cb52553 2023-03-20 10:57:51,005 DEBUG [Thread-2246] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:51,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:51,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:51,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:51,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:51,021 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:51,021 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:51,021 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:51,021 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:51,021 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:51,021 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:51,023 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:51,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:51,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:51,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:51,026 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,027 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,027 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:51,027 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:51,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:51,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:51,027 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:51,048 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:51,049 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:51,049 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:51,049 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:51,049 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:51,057 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:51,057 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:51,057 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:51,057 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. started... 2023-03-20 10:57:51,057 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. as already flushing 2023-03-20 10:57:51,057 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:51,060 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:51,061 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:51,061 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,061 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:51,061 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:51,062 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:51,062 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:51,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:51,062 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:51,062 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:51,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:51,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:51,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:51,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:51,064 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:51,064 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5cf7b489[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:51,064 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:51,064 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:51,066 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:51,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:51,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:51,066 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:51,066 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:51,066 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:51,066 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,066 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:51,082 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:51,082 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:51,082 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:51,082 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:51,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:51,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:51,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:51,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:51,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:51,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:51,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:51,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:51,109 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:51,109 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:51,109 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:51,109 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:51,109 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:51,110 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,111 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:51,111 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:51,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:51,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:51,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:51,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:51,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:51,112 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,113 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:51,113 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:51,113 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:51,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:51,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:51,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:51,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:51,114 DEBUG [Thread-2246] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:51,114 DEBUG [Thread-2246] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:51,114 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:51,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:51,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:51,114 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,114 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:51,114 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:51,114 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,114 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:51,114 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:51,114 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:51,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:51,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:51,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:51,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:51,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:51,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:51,115 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:57:51,176 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:51,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43494 deadline: 1679309931170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,176 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:51,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43530 deadline: 1679309931171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,191 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:51,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43528 deadline: 1679309931184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,192 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:51,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43492 deadline: 1679309931186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,215 DEBUG [Thread-2246] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:51,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:51,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:51,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:51,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:51,226 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:51,226 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:51,226 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:51,226 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:51,227 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:51,228 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:51,228 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,228 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:51,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:51,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:51,229 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,229 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:51,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:51,229 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:51,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:51,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:51,229 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:51,234 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:51,234 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:51,234 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:51,234 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:51,234 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:51,243 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:51,243 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:51,245 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:51,245 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. started... 2023-03-20 10:57:51,245 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. as already flushing 2023-03-20 10:57:51,245 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:51,245 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:51,245 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:51,245 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,245 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:51,245 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:51,249 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:51,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:51,249 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:51,249 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:51,249 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:51,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:51,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:51,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:51,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:51,251 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:51,251 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@886ef7f[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:51,251 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:51,251 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:51,252 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:51,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:51,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:51,252 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:51,252 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,253 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:51,252 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:51,253 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:51,254 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:51,254 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,254 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:51,254 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:51,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:51,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:51,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:51,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:51,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:51,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:51,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:51,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:51,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:51,264 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:51,264 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:51,264 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:51,264 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:51,264 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:51,270 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:51,270 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:51,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:51,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:51,270 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:51,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:51,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:51,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:51,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:51,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:51,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:51,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:51,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:51,275 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:51,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:51,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:51,276 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:51,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,279 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,279 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:51,279 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:51,279 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:51,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:51,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:51,279 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,279 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:51,279 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:51,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:51,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:51,279 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:51,279 DEBUG [Thread-2246] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:51,279 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:51,279 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:51,279 DEBUG [Thread-2246] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:51,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:51,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:51,380 DEBUG [Thread-2246] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:51,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:51,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:51,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:51,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:51,387 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:51,387 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:51,387 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:51,387 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:51,388 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:51,388 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:51,390 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:51,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:51,390 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:51,390 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,390 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:51,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:51,391 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:51,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:51,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:51,391 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:51,392 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:51,392 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:51,392 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:51,392 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:51,392 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:51,392 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:51,392 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:51,392 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:51,392 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. started... 2023-03-20 10:57:51,392 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. as already flushing 2023-03-20 10:57:51,392 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:51,392 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:51,392 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:51,392 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,392 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:51,392 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:51,393 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:51,394 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:51,394 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:51,396 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:51,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:51,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:51,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:51,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:51,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:51,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:51,398 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:51,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32a63d7f[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:51,398 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:51,398 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:51,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:51,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:51,399 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:51,399 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:51,399 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:51,399 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,399 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:51,399 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,399 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:51,399 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:51,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:51,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:51,400 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:51,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:51,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:51,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:51,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:51,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:51,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:51,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:51,404 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:51,404 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:51,404 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:51,404 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:51,404 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:51,406 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:51,406 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:51,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:51,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:51,406 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:51,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:51,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:51,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:51,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:51,408 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:51,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:51,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:51,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:51,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:51,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:51,411 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:51,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:51,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,414 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,414 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:51,414 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:51,414 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:51,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:51,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:51,414 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:57:51,414 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:51,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:51,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:51,414 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:51,414 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,414 DEBUG [Thread-2246] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:51,414 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:51,414 DEBUG [Thread-2246] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:51,414 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:51,414 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:51,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:51,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:51,466 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/B/dbb1dfcafa414400804ae42811f6cdc3 2023-03-20 10:57:51,505 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/C/fe6869de19cc482097839196c5a0695b 2023-03-20 10:57:51,510 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/A/1decfdfeb5f545388e5e811c1cb52553 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/1decfdfeb5f545388e5e811c1cb52553 2023-03-20 10:57:51,515 DEBUG [Thread-2246] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:51,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:51,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:51,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:51,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:51,520 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:51,520 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:51,520 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:51,520 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:51,520 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/1decfdfeb5f545388e5e811c1cb52553, entries=200, sequenceid=48, filesize=31.9 K 2023-03-20 10:57:51,521 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:51,521 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:51,522 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/B/dbb1dfcafa414400804ae42811f6cdc3 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/dbb1dfcafa414400804ae42811f6cdc3 2023-03-20 10:57:51,522 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:51,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:51,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:51,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:51,524 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,524 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:51,524 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,525 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:51,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:51,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:51,525 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:51,525 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:51,526 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:51,526 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:51,526 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:51,526 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:51,526 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:51,526 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:51,526 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:51,526 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. started... 2023-03-20 10:57:51,527 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. as already flushing 2023-03-20 10:57:51,527 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:51,527 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:51,527 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:51,527 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,527 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:51,527 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:51,528 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,528 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:51,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:51,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:51,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:51,532 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:51,532 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:51,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:51,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:51,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:51,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:51,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@690d9a0b[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:51,533 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:51,533 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:51,533 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:51,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:51,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:51,534 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:51,534 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:51,534 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:51,534 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,534 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:51,536 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/dbb1dfcafa414400804ae42811f6cdc3, entries=150, sequenceid=48, filesize=11.6 K 2023-03-20 10:57:51,537 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:51,537 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:51,537 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:51,537 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:51,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:51,538 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/C/fe6869de19cc482097839196c5a0695b as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/fe6869de19cc482097839196c5a0695b 2023-03-20 10:57:51,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:51,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:51,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:51,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:51,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:51,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:51,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:51,547 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:51,547 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:51,547 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:51,547 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:51,547 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:51,552 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/fe6869de19cc482097839196c5a0695b, entries=150, sequenceid=48, filesize=11.6 K 2023-03-20 10:57:51,553 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~361.50 KB/370176, currentSize=60.38 KB/61830 for ee10ffd90d917f64b63db6b593dd0991 in 1030ms, sequenceid=48, compaction requested=false 2023-03-20 10:57:51,553 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:51,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:51,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:51,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:51,553 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ee10ffd90d917f64b63db6b593dd0991: 2023-03-20 10:57:51,553 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:51,553 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:51,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:51,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:51,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:51,553 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:51,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:51,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:51,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:51,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:51,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:51,556 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:51,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:51,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,561 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,561 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:51,562 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:51,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:51,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:51,561 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:51,562 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:51,562 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,562 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:51,562 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:51,562 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:51,562 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:57:51,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:51,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:51,562 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:51,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:51,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:51,563 DEBUG [Thread-2246] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:51,563 DEBUG [Thread-2246] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:51,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on ee10ffd90d917f64b63db6b593dd0991 2023-03-20 10:57:51,590 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ee10ffd90d917f64b63db6b593dd0991 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-20 10:57:51,591 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee10ffd90d917f64b63db6b593dd0991, store=A 2023-03-20 10:57:51,591 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:51,591 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee10ffd90d917f64b63db6b593dd0991, store=B 2023-03-20 10:57:51,591 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:51,591 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee10ffd90d917f64b63db6b593dd0991, store=C 2023-03-20 10:57:51,591 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:51,663 DEBUG [Thread-2246] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:51,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:51,666 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:51,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:51,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:51,667 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:51,667 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:51,667 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:51,667 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:51,667 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:51,667 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:51,668 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:51,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:51,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:51,668 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:51,668 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,668 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:51,669 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:51,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:51,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:51,669 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:51,670 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:51,670 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:51,670 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:51,670 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:51,670 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:51,674 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:51,674 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:51,674 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:51,674 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. started... 2023-03-20 10:57:51,674 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. as already flushing 2023-03-20 10:57:51,674 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:51,674 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:51,674 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:51,674 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,674 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:51,674 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:51,678 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,678 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:51,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:51,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:51,679 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:51,679 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:51,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:51,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:51,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:51,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:51,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:51,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@83f285f[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:51,682 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:51,682 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:51,683 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:51,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:51,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:51,683 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:51,683 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:51,683 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:51,683 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,683 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:51,684 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:51,684 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:51,684 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:51,684 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:51,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:51,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:51,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:51,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:51,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:51,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:51,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:51,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:51,687 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:51,687 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:51,687 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:51,687 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:51,687 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:51,688 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:51,688 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:51,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:51,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:51,688 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:51,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:51,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:51,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:51,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:51,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:51,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:51,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:51,691 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:51,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:51,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:51,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,691 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:51,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:51,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,693 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,693 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:51,693 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:51,693 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:51,693 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:51,693 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,693 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:51,693 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:51,693 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:51,693 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:51,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:51,693 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:57:51,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:51,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:51,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:51,694 DEBUG [Thread-2246] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:51,694 DEBUG [Thread-2246] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:51,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:51,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:51,702 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:51,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43478 deadline: 1679309931690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,703 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:51,703 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:51,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43492 deadline: 1679309931694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43494 deadline: 1679309931696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,708 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:51,708 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:51,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43528 deadline: 1679309931699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43530 deadline: 1679309931703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,794 DEBUG [Thread-2246] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:51,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:51,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:51,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:51,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:51,797 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:51,797 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:51,797 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:51,797 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:51,797 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:51,797 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:51,798 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:51,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:51,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:51,798 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:51,799 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,799 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:51,799 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:51,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:51,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:51,800 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:51,800 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:51,800 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:51,800 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:51,800 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:51,800 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:51,800 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:51,800 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:51,800 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:51,800 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. started... 2023-03-20 10:57:51,800 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. as already flushing 2023-03-20 10:57:51,800 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:51,800 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:51,800 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:51,801 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,801 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:51,801 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:51,803 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:51,803 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:51,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:51,803 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:51,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:51,803 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:51,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:51,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:51,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:51,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:51,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37fdd330[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:51,805 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:51,805 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:51,810 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:51,810 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:51,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43494 deadline: 1679309931805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43478 deadline: 1679309931805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,811 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:51,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:51,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:51,811 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:51,811 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,811 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:51,811 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:51,811 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:51,812 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,812 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:51,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:51,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:51,812 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:51,812 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:51,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:51,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:51,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:51,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:51,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:51,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:51,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:51,814 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:51,814 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:51,814 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:51,814 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:51,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43530 deadline: 1679309931809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,814 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:51,814 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:51,815 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:51,815 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:51,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:51,815 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:51,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:51,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:51,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:51,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:51,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:51,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:51,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:51,816 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:51,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:51,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:51,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:51,817 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:51,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:51,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,818 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,818 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:51,818 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:51,818 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,818 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:51,818 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:51,818 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:51,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:51,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:51,818 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:57:51,818 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:51,818 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:51,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:51,819 DEBUG [Thread-2246] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:51,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:51,819 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:51,819 DEBUG [Thread-2246] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:51,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:51,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:51,919 DEBUG [Thread-2246] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:51,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:51,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:51,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:51,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:51,926 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:51,926 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:51,926 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:51,926 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:51,926 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:51,926 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:51,927 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,927 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:51,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:51,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:51,927 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,928 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:51,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:51,928 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:51,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:51,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:51,928 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:51,930 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:51,930 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:51,930 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:51,930 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:51,930 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:51,934 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:51,934 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:51,935 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:51,936 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. started... 2023-03-20 10:57:51,936 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. as already flushing 2023-03-20 10:57:51,936 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:51,936 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:51,936 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:51,936 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,936 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:51,936 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:51,937 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:51,937 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:51,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:51,937 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:51,937 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:51,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:51,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:51,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:51,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:51,940 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:51,940 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34334d75[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:51,940 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:51,940 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:51,941 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:51,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:51,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:51,941 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:51,941 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:51,941 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:51,942 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,942 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:51,942 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:51,942 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:51,942 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:51,942 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:51,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:51,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:51,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:51,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:51,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:51,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:51,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:51,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:51,950 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:51,950 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:51,950 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:51,950 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:51,950 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:51,953 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:51,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:51,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:51,953 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:51,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:51,953 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:51,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:51,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:51,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:51,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:51,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:51,955 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:51,956 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:51,962 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:51,962 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:51,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:51,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:51,962 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:51,962 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,962 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:51,962 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:51,962 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:51,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:51,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:51,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:51,962 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:51,962 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:51,962 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:51,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:51,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:51,963 DEBUG [Thread-2246] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:51,963 DEBUG [Thread-2246] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:51,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:51,964 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:57:52,016 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:52,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43478 deadline: 1679309932012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,018 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:52,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43494 deadline: 1679309932012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,020 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230320e66f3f0652c64d1295e0a6de0a86fe06 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320e66f3f0652c64d1295e0a6de0a86fe06 2023-03-20 10:57:52,023 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:52,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43530 deadline: 1679309932017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,032 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=61, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/A/6b2131cad89a471cb9149d29551e91e5 2023-03-20 10:57:52,063 DEBUG [Thread-2246] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:52,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:52,064 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=61 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/B/d54def83c41744209cde5c712c33fee4 2023-03-20 10:57:52,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:52,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:52,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:52,066 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:52,066 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:52,066 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:52,066 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:52,067 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:52,067 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:52,067 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:52,067 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:52,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:52,068 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,068 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:52,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:52,068 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:52,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:52,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:52,068 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:52,068 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:52,068 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:52,068 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:52,069 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:52,069 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:52,069 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:52,069 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:52,069 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:52,069 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. started... 2023-03-20 10:57:52,069 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. as already flushing 2023-03-20 10:57:52,069 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:52,069 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:52,069 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:52,069 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,069 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:52,069 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:52,070 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:52,070 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:52,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:52,070 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:52,070 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:52,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:52,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:52,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:52,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:52,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:52,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@316b543a[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:52,072 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:52,072 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:52,072 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:52,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:52,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:52,072 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:52,072 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:52,072 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:52,072 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,072 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:52,073 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,073 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:52,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:52,073 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:52,073 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:52,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:52,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:52,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:52,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:52,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:52,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:52,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:52,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:52,076 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:52,076 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:52,076 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:52,076 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:52,076 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:52,078 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:52,078 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:52,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:52,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:52,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:52,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:52,078 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:52,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:52,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:52,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:52,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:52,078 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:52,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:52,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:52,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:52,079 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:52,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:52,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,080 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:52,080 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:52,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:52,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:52,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:52,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:52,081 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,081 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:52,081 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:52,081 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,081 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:52,081 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:52,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:52,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:52,081 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:52,081 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:57:52,081 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:52,081 DEBUG [Thread-2246] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:52,081 DEBUG [Thread-2246] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:52,182 DEBUG [Thread-2246] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:52,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:52,184 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:52,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:52,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:52,185 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:52,185 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:52,185 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:52,185 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:52,186 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:52,186 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:52,186 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,187 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:52,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:52,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:52,188 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,188 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:52,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:52,190 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:52,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:52,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:52,191 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:52,191 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:52,191 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:52,191 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:52,191 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:52,191 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:52,191 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:52,191 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:52,191 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:52,191 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. started... 2023-03-20 10:57:52,192 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. as already flushing 2023-03-20 10:57:52,192 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:52,192 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:52,192 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:52,192 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,192 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:52,192 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:52,195 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,195 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:52,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:52,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:52,195 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:52,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:52,195 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:52,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:52,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:52,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:52,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:52,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4067ef98[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:52,196 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:52,196 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:52,197 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:52,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:52,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:52,197 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:52,197 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:52,197 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:52,197 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,197 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:52,199 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,199 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:52,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:52,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:52,199 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:52,199 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:52,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:52,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:52,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:52,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:52,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:52,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:52,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:52,201 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:52,202 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:52,202 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:52,202 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:52,202 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:52,203 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:52,203 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:52,203 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:52,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:52,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:52,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:52,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:52,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:52,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:52,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:52,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:52,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:52,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:52,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:52,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:52,206 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:52,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:52,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,207 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,209 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:52,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:52,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:52,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:52,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:52,209 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:52,209 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:52,209 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,209 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:52,209 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:52,209 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,209 DEBUG [Thread-2246] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:52,210 DEBUG [Thread-2246] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:52,209 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:52,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:52,210 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:52,210 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:52,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:52,310 DEBUG [Thread-2246] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:52,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:52,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:52,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:52,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:52,312 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:52,312 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:52,312 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:52,312 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:52,312 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:52,313 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:52,313 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,313 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:52,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:52,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:52,313 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,313 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:52,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:52,314 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:52,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:52,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:52,314 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:52,314 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:52,314 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:52,314 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:52,314 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:52,314 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:52,317 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:52,318 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:52,318 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:52,318 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. started... 2023-03-20 10:57:52,318 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. as already flushing 2023-03-20 10:57:52,318 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:52,318 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:52,318 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:52,318 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,318 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:52,318 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:52,319 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,319 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:52,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:52,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:52,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:52,319 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:52,319 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:52,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:52,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:52,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:52,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:52,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@195ac0f7[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:52,320 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:52,320 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:52,321 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:52,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:52,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:52,321 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:52,321 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,321 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:52,321 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:52,321 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:52,321 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:52,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43494 deadline: 1679309932319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,322 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:52,322 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:52,322 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:52,322 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:52,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43478 deadline: 1679309932321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,323 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:52,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:52,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:52,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:52,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:52,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:52,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:52,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:52,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:52,325 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:52,325 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:52,325 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:52,325 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:52,325 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:52,327 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:52,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43530 deadline: 1679309932325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,336 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:52,336 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:52,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:52,336 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:52,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:52,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:52,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:52,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:52,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:52,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:52,336 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:52,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:52,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:52,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:52,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:52,337 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:52,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:52,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,339 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:52,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:52,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:52,339 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:52,339 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:52,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:52,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:52,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:52,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:52,339 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:57:52,339 DEBUG [Thread-2246] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:52,339 DEBUG [Thread-2246] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:52,340 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,340 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:52,340 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:52,340 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,340 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:52,340 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:52,340 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:52,439 DEBUG [Thread-2246] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:52,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:52,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:52,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:52,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:52,442 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:52,442 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:52,442 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:52,442 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:52,442 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:52,442 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:52,443 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:52,443 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:52,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:52,443 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,443 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:52,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:52,443 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:52,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:52,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:52,444 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:52,444 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:52,444 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:52,444 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:52,444 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:52,444 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:52,444 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:52,444 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:52,444 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:52,444 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. started... 2023-03-20 10:57:52,444 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. as already flushing 2023-03-20 10:57:52,444 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:52,444 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:52,444 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:52,444 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,444 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:52,444 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:52,445 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,445 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:52,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:52,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:52,445 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:52,445 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:52,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:52,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:52,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:52,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:52,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:52,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37ac6e4c[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:52,447 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:52,447 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:52,448 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:52,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:52,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:52,448 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,448 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:52,448 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:52,448 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:52,448 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:52,448 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,449 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:52,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:52,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:52,449 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:52,449 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:52,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:52,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:52,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:52,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:52,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:52,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:52,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:52,450 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:52,450 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:52,450 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:52,450 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:52,450 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:52,451 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:52,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:52,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:52,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:52,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:52,451 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:52,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:52,452 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:52,452 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:52,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:52,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:52,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:52,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:52,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:52,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,452 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:52,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:52,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:52,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,454 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,454 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:52,454 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:52,454 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:52,454 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:52,454 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:57:52,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:52,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:52,454 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:52,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:52,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:52,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:52,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:52,454 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,454 DEBUG [Thread-2246] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:52,454 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:52,454 DEBUG [Thread-2246] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:52,454 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:52,454 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:52,484 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=61 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/C/aaf998f0e8cd4c3d8b2f1d13bec40a3f 2023-03-20 10:57:52,488 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/A/6b2131cad89a471cb9149d29551e91e5 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/6b2131cad89a471cb9149d29551e91e5 2023-03-20 10:57:52,495 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/6b2131cad89a471cb9149d29551e91e5, entries=150, sequenceid=61, filesize=25.1 K 2023-03-20 10:57:52,496 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/B/d54def83c41744209cde5c712c33fee4 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/d54def83c41744209cde5c712c33fee4 2023-03-20 10:57:52,505 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/d54def83c41744209cde5c712c33fee4, entries=100, sequenceid=61, filesize=9.3 K 2023-03-20 10:57:52,505 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/C/aaf998f0e8cd4c3d8b2f1d13bec40a3f as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/aaf998f0e8cd4c3d8b2f1d13bec40a3f 2023-03-20 10:57:52,517 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/aaf998f0e8cd4c3d8b2f1d13bec40a3f, entries=100, sequenceid=61, filesize=9.3 K 2023-03-20 10:57:52,519 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for ee10ffd90d917f64b63db6b593dd0991 in 929ms, sequenceid=61, compaction requested=true 2023-03-20 10:57:52,519 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ee10ffd90d917f64b63db6b593dd0991: 2023-03-20 10:57:52,519 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 10:57:52,519 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:57:52,520 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 10:57:52,520 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:57:52,520 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:57:52,520 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 91114 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 10:57:52,520 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 10:57:52,520 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1912): ee10ffd90d917f64b63db6b593dd0991/A is initiating minor compaction (all files) 2023-03-20 10:57:52,520 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): ee10ffd90d917f64b63db6b593dd0991/B is initiating minor compaction (all files) 2023-03-20 10:57:52,520 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ee10ffd90d917f64b63db6b593dd0991/A in TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:52,520 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ee10ffd90d917f64b63db6b593dd0991/B in TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:52,521 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/8167b7abdc364b828f2f5a16e552b1b5, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/dbb1dfcafa414400804ae42811f6cdc3, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/d54def83c41744209cde5c712c33fee4] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp, totalSize=32.5 K 2023-03-20 10:57:52,521 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/0602a43a94cd4cdf880607998644fb04, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/1decfdfeb5f545388e5e811c1cb52553, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/6b2131cad89a471cb9149d29551e91e5] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp, totalSize=89.0 K 2023-03-20 10:57:52,521 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 8167b7abdc364b828f2f5a16e552b1b5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1679309869367 2023-03-20 10:57:52,521 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 0602a43a94cd4cdf880607998644fb04, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1679309869367 2023-03-20 10:57:52,521 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting dbb1dfcafa414400804ae42811f6cdc3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1679309869414 2023-03-20 10:57:52,521 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 1decfdfeb5f545388e5e811c1cb52553, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1679309869414 2023-03-20 10:57:52,521 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting d54def83c41744209cde5c712c33fee4, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1679309870544 2023-03-20 10:57:52,521 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 6b2131cad89a471cb9149d29551e91e5, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1679309870544 2023-03-20 10:57:52,529 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): ee10ffd90d917f64b63db6b593dd0991#B#compaction#306 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:57:52,537 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] throttle.PressureAwareThroughputController(145): ee10ffd90d917f64b63db6b593dd0991#A#compaction#307 average throughput is 2.19 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:57:52,558 DEBUG [Thread-2246] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:52,560 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/B/176eb5c60819464f876e82e90e3ba7a4 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/176eb5c60819464f876e82e90e3ba7a4 2023-03-20 10:57:52,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:52,565 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ee10ffd90d917f64b63db6b593dd0991/B of ee10ffd90d917f64b63db6b593dd0991 into 176eb5c60819464f876e82e90e3ba7a4(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:57:52,565 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ee10ffd90d917f64b63db6b593dd0991: 2023-03-20 10:57:52,565 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991., storeName=ee10ffd90d917f64b63db6b593dd0991/B, priority=13, startTime=1679309872519; duration=0sec 2023-03-20 10:57:52,565 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:57:52,565 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 10:57:52,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:52,567 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 10:57:52,567 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): ee10ffd90d917f64b63db6b593dd0991/C is initiating minor compaction (all files) 2023-03-20 10:57:52,567 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ee10ffd90d917f64b63db6b593dd0991/C in TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:52,567 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/00f4ffb13fb54e29b53d38a691eadd1c, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/fe6869de19cc482097839196c5a0695b, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/aaf998f0e8cd4c3d8b2f1d13bec40a3f] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp, totalSize=32.5 K 2023-03-20 10:57:52,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:52,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:52,570 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:52,570 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:52,570 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:52,570 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:52,571 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:52,571 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:52,572 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 00f4ffb13fb54e29b53d38a691eadd1c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1679309869367 2023-03-20 10:57:52,573 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting fe6869de19cc482097839196c5a0695b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1679309869414 2023-03-20 10:57:52,573 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,574 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,574 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:52,574 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting aaf998f0e8cd4c3d8b2f1d13bec40a3f, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1679309870544 2023-03-20 10:57:52,578 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:52,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:52,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:52,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:52,582 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:52,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:52,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:52,582 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:52,606 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:52,606 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:52,606 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:52,606 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:52,606 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:52,620 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:52,620 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:52,620 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:52,620 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. started... 2023-03-20 10:57:52,620 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ee10ffd90d917f64b63db6b593dd0991 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-20 10:57:52,620 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee10ffd90d917f64b63db6b593dd0991, store=A 2023-03-20 10:57:52,620 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:52,620 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee10ffd90d917f64b63db6b593dd0991, store=B 2023-03-20 10:57:52,620 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:52,620 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee10ffd90d917f64b63db6b593dd0991, store=C 2023-03-20 10:57:52,621 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:52,622 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): ee10ffd90d917f64b63db6b593dd0991#C#compaction#308 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:57:52,690 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/C/dd2696344d8442a79ac6f57f49b6720e as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/dd2696344d8442a79ac6f57f49b6720e 2023-03-20 10:57:52,696 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ee10ffd90d917f64b63db6b593dd0991/C of ee10ffd90d917f64b63db6b593dd0991 into dd2696344d8442a79ac6f57f49b6720e(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:57:52,696 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ee10ffd90d917f64b63db6b593dd0991: 2023-03-20 10:57:52,696 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991., storeName=ee10ffd90d917f64b63db6b593dd0991/C, priority=13, startTime=1679309872520; duration=0sec 2023-03-20 10:57:52,697 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:57:52,711 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. as already flushing 2023-03-20 10:57:52,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on ee10ffd90d917f64b63db6b593dd0991 2023-03-20 10:57:52,713 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230320d5ad38daba754cb7bb99afda409a970d to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320d5ad38daba754cb7bb99afda409a970d 2023-03-20 10:57:52,735 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=81, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/A/f46475f61d6b46bb82b447f2cb0a96f8 2023-03-20 10:57:52,751 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:52,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43492 deadline: 1679309932742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,755 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:52,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43528 deadline: 1679309932750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,829 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:52,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43494 deadline: 1679309932826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,830 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:52,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43478 deadline: 1679309932826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,833 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:52,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43530 deadline: 1679309932829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,861 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:52,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43492 deadline: 1679309932855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:52,866 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:52,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43528 deadline: 1679309932860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:53,066 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:53,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43492 deadline: 1679309933063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:53,071 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:53,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43528 deadline: 1679309933067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:53,146 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=81 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/B/369da498a1d949f1a4305a3b881de17d 2023-03-20 10:57:53,154 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=81 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/C/edee06c383124e21bb308afaf7bbf239 2023-03-20 10:57:53,158 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/A/f46475f61d6b46bb82b447f2cb0a96f8 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/f46475f61d6b46bb82b447f2cb0a96f8 2023-03-20 10:57:53,161 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/f46475f61d6b46bb82b447f2cb0a96f8, entries=150, sequenceid=81, filesize=25.1 K 2023-03-20 10:57:53,161 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/B/369da498a1d949f1a4305a3b881de17d as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/369da498a1d949f1a4305a3b881de17d 2023-03-20 10:57:53,164 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/369da498a1d949f1a4305a3b881de17d, entries=150, sequenceid=81, filesize=11.6 K 2023-03-20 10:57:53,164 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.StoreScanner(953): StoreScanner already has the close lock. There is no need to updateReaders 2023-03-20 10:57:53,165 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/C/edee06c383124e21bb308afaf7bbf239 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/edee06c383124e21bb308afaf7bbf239 2023-03-20 10:57:53,168 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/edee06c383124e21bb308afaf7bbf239, entries=150, sequenceid=81, filesize=11.6 K 2023-03-20 10:57:53,169 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for ee10ffd90d917f64b63db6b593dd0991 in 549ms, sequenceid=81, compaction requested=false 2023-03-20 10:57:53,169 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ee10ffd90d917f64b63db6b593dd0991: 2023-03-20 10:57:53,169 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:53,169 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:53,169 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:53,169 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:53,169 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:53,169 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:53,170 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:53,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:53,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:53,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:53,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:53,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:53,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:53,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:53,171 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:53,172 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:53,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:53,172 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:53,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:53,172 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:53,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c8d2607[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:53,172 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:53,173 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:53,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:53,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:53,173 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:53,173 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:53,173 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:53,173 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:53,173 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:53,173 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:53,173 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:53,174 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:53,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:53,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:53,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:53,174 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:53,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:53,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:53,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:53,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:53,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:53,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:53,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:53,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:53,177 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:53,177 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:53,177 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:53,177 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:53,177 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:53,177 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:53,179 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:53,179 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:53,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:53,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:53,180 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:53,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:53,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:53,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:53,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:53,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:53,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:53,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:53,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:53,181 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:53,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:53,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:53,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:53,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:53,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:53,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:53,183 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:53,187 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:53,187 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:53,187 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:53,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:53,188 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:53,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:53,188 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:53,188 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:53,188 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:53,188 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:53,188 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:53,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:53,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:53,190 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:53,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:53,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:53,190 DEBUG [Thread-2246] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:53,190 DEBUG [Thread-2246] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:53,291 DEBUG [Thread-2246] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:53,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:53,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:53,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:53,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:53,293 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:53,293 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:53,293 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:53,293 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:53,302 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:53,302 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:53,303 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:53,303 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:53,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:53,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:53,303 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:53,303 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:53,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:53,304 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:53,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:53,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:53,305 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:53,305 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:53,305 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:53,305 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:53,305 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:53,305 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:53,306 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:53,306 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:53,306 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:53,306 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. started... 2023-03-20 10:57:53,306 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ee10ffd90d917f64b63db6b593dd0991 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-20 10:57:53,306 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee10ffd90d917f64b63db6b593dd0991, store=A 2023-03-20 10:57:53,306 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:53,306 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee10ffd90d917f64b63db6b593dd0991, store=B 2023-03-20 10:57:53,306 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:53,306 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee10ffd90d917f64b63db6b593dd0991, store=C 2023-03-20 10:57:53,306 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:53,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on ee10ffd90d917f64b63db6b593dd0991 2023-03-20 10:57:53,372 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. as already flushing 2023-03-20 10:57:53,418 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/A/82c0e3e07fd54edab8508e6d284270a6 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/82c0e3e07fd54edab8508e6d284270a6 2023-03-20 10:57:53,424 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ee10ffd90d917f64b63db6b593dd0991/A of ee10ffd90d917f64b63db6b593dd0991 into 82c0e3e07fd54edab8508e6d284270a6(size=25.2 K), total size for store is 50.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:57:53,424 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ee10ffd90d917f64b63db6b593dd0991: 2023-03-20 10:57:53,424 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991., storeName=ee10ffd90d917f64b63db6b593dd0991/A, priority=13, startTime=1679309872519; duration=0sec 2023-03-20 10:57:53,424 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:57:53,430 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:53,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43492 deadline: 1679309933425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:53,433 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:53,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43528 deadline: 1679309933427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:53,536 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:53,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43492 deadline: 1679309933531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:53,540 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:53,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43528 deadline: 1679309933534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:53,729 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230320b16b889a7c7142e1b14c50175ea20cbb to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320b16b889a7c7142e1b14c50175ea20cbb 2023-03-20 10:57:53,741 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:53,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43492 deadline: 1679309933737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:53,747 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:53,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43528 deadline: 1679309933742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:53,839 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:53,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43478 deadline: 1679309933835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:53,839 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:53,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43494 deadline: 1679309933836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:53,846 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:53,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43530 deadline: 1679309933843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:54,051 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:54,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43492 deadline: 1679309934043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:54,054 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:54,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43528 deadline: 1679309934048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:54,134 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=95, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/A/14a11b7bfca34fecbfe24856b5d70122 2023-03-20 10:57:54,145 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=95 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/B/ab9fe79b9e2d438db36565392a48e7ea 2023-03-20 10:57:54,162 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=95 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/C/e48c013af12f4b559b28431c77989c96 2023-03-20 10:57:54,167 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/A/14a11b7bfca34fecbfe24856b5d70122 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/14a11b7bfca34fecbfe24856b5d70122 2023-03-20 10:57:54,175 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/14a11b7bfca34fecbfe24856b5d70122, entries=150, sequenceid=95, filesize=25.1 K 2023-03-20 10:57:54,176 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/B/ab9fe79b9e2d438db36565392a48e7ea as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/ab9fe79b9e2d438db36565392a48e7ea 2023-03-20 10:57:54,186 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/ab9fe79b9e2d438db36565392a48e7ea, entries=150, sequenceid=95, filesize=11.6 K 2023-03-20 10:57:54,186 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/C/e48c013af12f4b559b28431c77989c96 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/e48c013af12f4b559b28431c77989c96 2023-03-20 10:57:54,193 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/e48c013af12f4b559b28431c77989c96, entries=150, sequenceid=95, filesize=11.6 K 2023-03-20 10:57:54,194 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for ee10ffd90d917f64b63db6b593dd0991 in 888ms, sequenceid=95, compaction requested=true 2023-03-20 10:57:54,194 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ee10ffd90d917f64b63db6b593dd0991: 2023-03-20 10:57:54,194 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:54,194 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:54,194 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:54,194 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:54,194 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:54,195 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:54,195 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:54,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:54,196 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:54,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:54,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:54,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:54,196 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:54,196 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:54,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:54,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:54,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:54,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:54,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:54,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23690921[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:54,198 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:54,198 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:54,199 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:54,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:54,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:54,199 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:54,199 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:54,199 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:54,199 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:54,199 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:54,200 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:54,200 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:54,200 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:54,201 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:54,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:54,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:54,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:54,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:54,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:54,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:54,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:54,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:54,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:54,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:54,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:54,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:54,215 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:54,215 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:54,215 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:54,215 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:54,215 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:54,217 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:54,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:54,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:54,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:54,217 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:54,217 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:54,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:54,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:54,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:54,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:54,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:54,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:54,218 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:54,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:54,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:54,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:54,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:54,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:54,219 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:54,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:54,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:54,222 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:54,222 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:54,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:54,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:54,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:54,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:54,222 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:54,222 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:54,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:54,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:54,222 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:54,222 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:54,223 DEBUG [Thread-2246] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:54,223 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:54,223 DEBUG [Thread-2246] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:54,223 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:54,223 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:54,223 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:54,323 DEBUG [Thread-2246] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:54,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:54,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:54,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:54,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:54,327 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:54,327 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:54,327 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:54,327 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:54,328 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:54,328 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:54,329 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:54,329 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:54,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:54,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:54,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:54,331 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:54,331 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:54,331 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:54,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:54,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:54,332 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:54,332 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:54,332 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:54,332 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:54,332 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:54,332 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:54,332 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:54,332 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:54,332 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:54,332 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. started... 2023-03-20 10:57:54,332 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ee10ffd90d917f64b63db6b593dd0991 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-20 10:57:54,333 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee10ffd90d917f64b63db6b593dd0991, store=A 2023-03-20 10:57:54,333 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:54,333 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee10ffd90d917f64b63db6b593dd0991, store=B 2023-03-20 10:57:54,333 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:54,333 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee10ffd90d917f64b63db6b593dd0991, store=C 2023-03-20 10:57:54,333 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:54,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on ee10ffd90d917f64b63db6b593dd0991 2023-03-20 10:57:54,561 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. as already flushing 2023-03-20 10:57:54,604 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:54,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43492 deadline: 1679309934596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:54,610 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:54,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43528 deadline: 1679309934603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:54,713 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:54,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43492 deadline: 1679309934706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:54,716 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:54,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43528 deadline: 1679309934712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:54,768 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230320dd46ad220d2043bdac1436c307c418e5 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320dd46ad220d2043bdac1436c307c418e5 2023-03-20 10:57:54,830 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=115, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/A/c1d27d0a757442b1b9be344d34fe2819 2023-03-20 10:57:54,844 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=115 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/B/c863e9d7ff39436ebfdb7610eb0c0040 2023-03-20 10:57:54,885 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=115 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/C/de865513902140438491cb469b37f997 2023-03-20 10:57:54,891 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/A/c1d27d0a757442b1b9be344d34fe2819 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/c1d27d0a757442b1b9be344d34fe2819 2023-03-20 10:57:54,895 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/c1d27d0a757442b1b9be344d34fe2819, entries=150, sequenceid=115, filesize=25.1 K 2023-03-20 10:57:54,896 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/B/c863e9d7ff39436ebfdb7610eb0c0040 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/c863e9d7ff39436ebfdb7610eb0c0040 2023-03-20 10:57:54,905 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/c863e9d7ff39436ebfdb7610eb0c0040, entries=150, sequenceid=115, filesize=11.6 K 2023-03-20 10:57:54,906 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/C/de865513902140438491cb469b37f997 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/de865513902140438491cb469b37f997 2023-03-20 10:57:54,911 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/de865513902140438491cb469b37f997, entries=150, sequenceid=115, filesize=11.6 K 2023-03-20 10:57:54,911 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for ee10ffd90d917f64b63db6b593dd0991 in 579ms, sequenceid=115, compaction requested=true 2023-03-20 10:57:54,911 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ee10ffd90d917f64b63db6b593dd0991: 2023-03-20 10:57:54,911 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:54,911 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:54,912 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:54,912 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:54,916 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:54,917 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:54,917 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:54,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:54,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:54,918 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:54,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:54,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:54,918 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:54,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:54,918 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:54,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:54,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:54,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:54,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:54,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c7b28c3[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:54,919 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:54,920 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:54,920 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:54,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:54,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:54,920 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:54,920 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:54,920 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:54,920 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:54,920 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:54,921 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:54,921 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:54,921 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:54,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:54,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:54,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:54,921 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:54,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:54,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:54,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:54,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:54,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:54,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:54,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:54,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:54,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:54,927 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:54,927 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:54,927 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:54,927 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:54,927 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:54,929 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:54,929 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:54,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:54,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:54,929 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:54,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:54,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:54,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:54,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:54,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:54,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:54,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:54,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:54,930 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:54,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:54,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:54,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:54,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:54,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:54,931 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:54,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:54,934 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:54,934 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:54,934 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:54,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:54,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:54,934 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:54,934 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:54,934 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:54,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:54,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:54,934 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:54,934 DEBUG [Thread-2246] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:54,934 DEBUG [Thread-2246] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:54,934 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:54,934 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:54,934 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:54,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:54,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:54,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on ee10ffd90d917f64b63db6b593dd0991 2023-03-20 10:57:54,959 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ee10ffd90d917f64b63db6b593dd0991 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-20 10:57:54,959 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee10ffd90d917f64b63db6b593dd0991, store=A 2023-03-20 10:57:54,960 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:54,960 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee10ffd90d917f64b63db6b593dd0991, store=B 2023-03-20 10:57:54,960 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:54,960 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee10ffd90d917f64b63db6b593dd0991, store=C 2023-03-20 10:57:54,960 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:54,981 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303206e2f118987c04cfaa6cba9f44d943802 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303206e2f118987c04cfaa6cba9f44d943802 2023-03-20 10:57:54,986 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=129, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/A/8f8f523d000247678ba5c8e3056b08fc 2023-03-20 10:57:55,034 DEBUG [Thread-2246] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:55,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:55,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:55,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:55,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:55,048 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:55,048 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:55,048 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:55,048 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:55,058 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:55,058 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:55,060 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:55,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43492 deadline: 1679309935055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:55,062 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:55,062 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:55,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:55,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:55,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:55,064 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:55,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:55,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:55,065 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:55,065 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:55,066 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:55,071 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:55,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43528 deadline: 1679309935061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:55,082 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:55,082 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:55,082 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:55,082 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:55,082 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:55,090 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:55,090 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:55,090 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:55,090 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. started... 2023-03-20 10:57:55,090 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. as already flushing 2023-03-20 10:57:55,090 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:55,094 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:55,094 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:55,094 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:55,095 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:55,095 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:55,096 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:55,096 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:55,097 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:55,097 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:55,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:55,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:55,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:55,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:55,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:55,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:55,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:55,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:55,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:55,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38f1f50e[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:55,102 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:55,103 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:55,104 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:55,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:55,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:55,104 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:55,104 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:55,104 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:55,104 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:55,104 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:55,105 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:55,105 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:55,105 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:55,105 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:55,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:55,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:55,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:55,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:55,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:55,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:55,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:55,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:55,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:55,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:55,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:55,107 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:55,107 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:55,107 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:55,107 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:55,107 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:55,108 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:55,109 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:55,109 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:55,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:55,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:55,109 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:55,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:55,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:55,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:55,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:55,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:55,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:55,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:55,110 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:55,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:55,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:55,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:55,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:55,112 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:55,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:55,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:55,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:55,121 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:55,121 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:55,122 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:55,122 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:55,122 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:55,122 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:55,122 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:55,122 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:55,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:55,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:55,122 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:55,122 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:55,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:55,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:55,122 DEBUG [Thread-2246] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:55,122 DEBUG [Thread-2246] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:55,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:55,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:55,166 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:55,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43492 deadline: 1679309935162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:55,177 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:55,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43528 deadline: 1679309935172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:55,222 DEBUG [Thread-2246] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:55,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:55,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:55,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:55,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:55,226 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:55,226 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:55,226 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:55,226 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:55,239 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:55,239 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:55,240 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:55,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:55,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:55,240 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:55,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:55,240 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:55,241 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:55,241 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:55,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:55,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:55,241 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:55,241 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:55,241 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:55,241 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:55,241 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:55,242 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:55,250 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:55,250 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:55,254 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:55,254 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. started... 2023-03-20 10:57:55,254 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. as already flushing 2023-03-20 10:57:55,254 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:55,254 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:55,254 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:55,254 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:55,254 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:55,254 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:55,256 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:55,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:55,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:55,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:55,256 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:55,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:55,256 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:55,256 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:55,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:55,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:55,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:55,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:55,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:55,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@df95109[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:55,258 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:55,258 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:55,258 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:55,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:55,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:55,259 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:55,259 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:55,259 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:55,259 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:55,259 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:55,259 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:55,260 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:55,260 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:55,260 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:55,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:55,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:55,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:55,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:55,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:55,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:55,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:55,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:55,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:55,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:55,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:55,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:55,262 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:55,262 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:55,262 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:55,262 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:55,262 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:55,263 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:55,263 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:55,263 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:55,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:55,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:55,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:55,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:55,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:55,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:55,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:55,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:55,264 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:55,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:55,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:55,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:55,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:55,267 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:55,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:55,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:55,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:55,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:55,279 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:55,279 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:55,279 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:55,279 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:55,279 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:55,279 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:55,279 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:55,279 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:55,279 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:55,280 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:55,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:55,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:55,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:55,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:55,280 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:57:55,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:55,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:55,284 DEBUG [Thread-2246] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:55,284 DEBUG [Thread-2246] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:55,374 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:55,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43492 deadline: 1679309935367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:55,383 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:55,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43528 deadline: 1679309935379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:55,384 DEBUG [Thread-2246] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:55,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:55,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:55,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:55,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:55,394 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:55,394 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:55,394 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:55,394 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:55,395 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:55,395 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:55,397 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:55,397 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:55,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:55,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:55,398 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:55,398 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=129 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/B/957a683fe95c49a8949b86da2763ca16 2023-03-20 10:57:55,398 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:55,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:55,399 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:55,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:55,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:55,400 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:55,400 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:55,400 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:55,400 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:55,400 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:55,400 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:55,413 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:55,413 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:55,413 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:55,413 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. started... 2023-03-20 10:57:55,413 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. as already flushing 2023-03-20 10:57:55,413 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:55,413 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:55,413 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:55,414 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:55,414 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:55,414 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:55,415 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:55,415 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:55,415 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:55,415 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:55,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:55,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:55,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:55,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:55,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:55,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:55,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:55,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:55,418 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:55,418 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66b26196[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:55,418 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:55,418 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:55,419 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:55,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:55,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:55,419 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:55,419 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:55,419 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:55,420 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:55,420 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:55,420 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:55,420 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:55,420 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:55,420 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:55,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:55,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:55,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:55,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:55,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:55,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:55,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:55,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:55,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:55,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:55,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:55,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:55,425 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:55,425 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:55,425 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:55,425 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:55,425 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:55,429 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:55,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:55,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:55,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:55,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:55,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:55,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:55,430 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:55,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:55,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:55,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:55,430 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:55,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:55,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:55,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:55,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:55,431 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:55,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:55,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:55,433 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:55,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:55,437 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:55,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:55,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:55,437 DEBUG [Thread-2246] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:55,437 DEBUG [Thread-2246] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:55,437 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:55,437 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:55,437 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:55,438 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:55,438 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:55,438 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:55,438 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:55,438 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:57:55,438 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:55,438 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:55,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:55,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:55,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:55,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:55,445 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=129 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/C/e6d6072d1f194b01858b0258e058c3cf 2023-03-20 10:57:55,451 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/A/8f8f523d000247678ba5c8e3056b08fc as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/8f8f523d000247678ba5c8e3056b08fc 2023-03-20 10:57:55,456 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/8f8f523d000247678ba5c8e3056b08fc, entries=150, sequenceid=129, filesize=25.2 K 2023-03-20 10:57:55,460 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/B/957a683fe95c49a8949b86da2763ca16 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/957a683fe95c49a8949b86da2763ca16 2023-03-20 10:57:55,472 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/957a683fe95c49a8949b86da2763ca16, entries=150, sequenceid=129, filesize=11.7 K 2023-03-20 10:57:55,474 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/C/e6d6072d1f194b01858b0258e058c3cf as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/e6d6072d1f194b01858b0258e058c3cf 2023-03-20 10:57:55,478 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/e6d6072d1f194b01858b0258e058c3cf, entries=150, sequenceid=129, filesize=11.7 K 2023-03-20 10:57:55,479 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for ee10ffd90d917f64b63db6b593dd0991 in 519ms, sequenceid=129, compaction requested=true 2023-03-20 10:57:55,479 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ee10ffd90d917f64b63db6b593dd0991: 2023-03-20 10:57:55,479 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:57:55,479 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-20 10:57:55,479 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-20 10:57:55,479 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:57:55,479 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:57:55,480 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 128883 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-20 10:57:55,480 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1912): ee10ffd90d917f64b63db6b593dd0991/A is initiating minor compaction (all files) 2023-03-20 10:57:55,480 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59583 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-20 10:57:55,480 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ee10ffd90d917f64b63db6b593dd0991/A in TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:55,481 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): ee10ffd90d917f64b63db6b593dd0991/B is initiating minor compaction (all files) 2023-03-20 10:57:55,481 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ee10ffd90d917f64b63db6b593dd0991/B in TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:55,481 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/82c0e3e07fd54edab8508e6d284270a6, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/f46475f61d6b46bb82b447f2cb0a96f8, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/14a11b7bfca34fecbfe24856b5d70122, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/c1d27d0a757442b1b9be344d34fe2819, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/8f8f523d000247678ba5c8e3056b08fc] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp, totalSize=125.9 K 2023-03-20 10:57:55,481 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/176eb5c60819464f876e82e90e3ba7a4, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/369da498a1d949f1a4305a3b881de17d, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/ab9fe79b9e2d438db36565392a48e7ea, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/c863e9d7ff39436ebfdb7610eb0c0040, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/957a683fe95c49a8949b86da2763ca16] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp, totalSize=58.2 K 2023-03-20 10:57:55,481 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 82c0e3e07fd54edab8508e6d284270a6, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1679309870522 2023-03-20 10:57:55,481 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 176eb5c60819464f876e82e90e3ba7a4, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1679309870522 2023-03-20 10:57:55,481 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting f46475f61d6b46bb82b447f2cb0a96f8, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=81, earliestPutTs=1679309871680 2023-03-20 10:57:55,482 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 369da498a1d949f1a4305a3b881de17d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=81, earliestPutTs=1679309871680 2023-03-20 10:57:55,482 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 14a11b7bfca34fecbfe24856b5d70122, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1679309872738 2023-03-20 10:57:55,483 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting c1d27d0a757442b1b9be344d34fe2819, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=115, earliestPutTs=1679309873414 2023-03-20 10:57:55,483 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting ab9fe79b9e2d438db36565392a48e7ea, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1679309872738 2023-03-20 10:57:55,484 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 8f8f523d000247678ba5c8e3056b08fc, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=129, earliestPutTs=1679309874594 2023-03-20 10:57:55,484 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting c863e9d7ff39436ebfdb7610eb0c0040, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=115, earliestPutTs=1679309873414 2023-03-20 10:57:55,485 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 957a683fe95c49a8949b86da2763ca16, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=129, earliestPutTs=1679309874594 2023-03-20 10:57:55,500 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): ee10ffd90d917f64b63db6b593dd0991#B#compaction#321 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:57:55,535 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] throttle.PressureAwareThroughputController(145): ee10ffd90d917f64b63db6b593dd0991#A#compaction#322 average throughput is 0.86 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:57:55,537 DEBUG [Thread-2246] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:55,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:55,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:55,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:55,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:55,542 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:55,542 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:55,542 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:55,542 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:55,543 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/B/d81acef791404fc09c434ff0ea87e8f0 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/d81acef791404fc09c434ff0ea87e8f0 2023-03-20 10:57:55,543 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:55,543 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:55,544 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:55,544 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:55,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:55,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:55,544 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:55,544 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:55,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:55,545 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:55,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:55,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:55,545 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:55,550 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in ee10ffd90d917f64b63db6b593dd0991/B of ee10ffd90d917f64b63db6b593dd0991 into d81acef791404fc09c434ff0ea87e8f0(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:57:55,551 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ee10ffd90d917f64b63db6b593dd0991: 2023-03-20 10:57:55,551 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991., storeName=ee10ffd90d917f64b63db6b593dd0991/B, priority=11, startTime=1679309875479; duration=0sec 2023-03-20 10:57:55,551 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:57:55,551 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-20 10:57:55,552 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59583 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-20 10:57:55,552 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:55,553 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): ee10ffd90d917f64b63db6b593dd0991/C is initiating minor compaction (all files) 2023-03-20 10:57:55,553 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:55,553 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:55,553 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:55,553 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ee10ffd90d917f64b63db6b593dd0991/C in TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:55,553 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:55,553 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/dd2696344d8442a79ac6f57f49b6720e, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/edee06c383124e21bb308afaf7bbf239, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/e48c013af12f4b559b28431c77989c96, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/de865513902140438491cb469b37f997, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/e6d6072d1f194b01858b0258e058c3cf] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp, totalSize=58.2 K 2023-03-20 10:57:55,553 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting dd2696344d8442a79ac6f57f49b6720e, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1679309870522 2023-03-20 10:57:55,554 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting edee06c383124e21bb308afaf7bbf239, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=81, earliestPutTs=1679309871680 2023-03-20 10:57:55,554 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting e48c013af12f4b559b28431c77989c96, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1679309872738 2023-03-20 10:57:55,555 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting de865513902140438491cb469b37f997, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=115, earliestPutTs=1679309873414 2023-03-20 10:57:55,555 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting e6d6072d1f194b01858b0258e058c3cf, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=129, earliestPutTs=1679309874594 2023-03-20 10:57:55,590 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:55,590 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:55,591 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:55,592 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. started... 2023-03-20 10:57:55,592 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ee10ffd90d917f64b63db6b593dd0991 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-20 10:57:55,592 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee10ffd90d917f64b63db6b593dd0991, store=A 2023-03-20 10:57:55,592 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:55,592 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee10ffd90d917f64b63db6b593dd0991, store=B 2023-03-20 10:57:55,592 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:55,592 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee10ffd90d917f64b63db6b593dd0991, store=C 2023-03-20 10:57:55,592 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:55,612 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): ee10ffd90d917f64b63db6b593dd0991#C#compaction#323 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:57:55,666 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/C/08c958c21ecb4751a10a92ffdbe1feaa as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/08c958c21ecb4751a10a92ffdbe1feaa 2023-03-20 10:57:55,675 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in ee10ffd90d917f64b63db6b593dd0991/C of ee10ffd90d917f64b63db6b593dd0991 into 08c958c21ecb4751a10a92ffdbe1feaa(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:57:55,675 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ee10ffd90d917f64b63db6b593dd0991: 2023-03-20 10:57:55,675 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991., storeName=ee10ffd90d917f64b63db6b593dd0991/C, priority=11, startTime=1679309875479; duration=0sec 2023-03-20 10:57:55,675 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:57:55,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on ee10ffd90d917f64b63db6b593dd0991 2023-03-20 10:57:55,680 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. as already flushing 2023-03-20 10:57:55,709 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:55,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43528 deadline: 1679309935702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:55,709 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:55,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43492 deadline: 1679309935706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:55,820 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:55,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43492 deadline: 1679309935810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:55,820 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:55,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43528 deadline: 1679309935810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:55,850 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:55,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43494 deadline: 1679309935845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:55,851 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:55,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43478 deadline: 1679309935848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:55,852 DEBUG [Thread-2236] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4156 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991., hostname=jenkins-hbase20.apache.org,36041,1679309777062, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-20 10:57:55,855 DEBUG [Thread-2238] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4162 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991., hostname=jenkins-hbase20.apache.org,36041,1679309777062, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-20 10:57:55,865 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:55,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43530 deadline: 1679309935863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:55,867 DEBUG [Thread-2240] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4162 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991., hostname=jenkins-hbase20.apache.org,36041,1679309777062, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-20 10:57:56,028 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:56,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43528 deadline: 1679309936022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:56,028 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:56,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43492 deadline: 1679309936022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:56,070 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303200b34e45923fd4109975e03b00de8166e to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303200b34e45923fd4109975e03b00de8166e 2023-03-20 10:57:56,337 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:56,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43492 deadline: 1679309936329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:56,338 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:56,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43528 deadline: 1679309936330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:56,504 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=148, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/A/026e32c9064f48e792c69f902a7c5ca6 2023-03-20 10:57:56,842 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/A/d496e90bb995417c9b71d15bafb8fc99 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/d496e90bb995417c9b71d15bafb8fc99 2023-03-20 10:57:56,848 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in ee10ffd90d917f64b63db6b593dd0991/A of ee10ffd90d917f64b63db6b593dd0991 into d496e90bb995417c9b71d15bafb8fc99(size=25.5 K), total size for store is 25.5 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-03-20 10:57:56,849 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ee10ffd90d917f64b63db6b593dd0991: 2023-03-20 10:57:56,849 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991., storeName=ee10ffd90d917f64b63db6b593dd0991/A, priority=11, startTime=1679309875479; duration=1sec 2023-03-20 10:57:56,849 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:57:56,851 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:56,851 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:56,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43492 deadline: 1679309936845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:56,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43528 deadline: 1679309936845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:56,919 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=148 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/B/6708edf3beec490ab0550473db651135 2023-03-20 10:57:56,933 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=148 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/C/a19007863ea047cba240f46626498f7b 2023-03-20 10:57:56,938 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/A/026e32c9064f48e792c69f902a7c5ca6 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/026e32c9064f48e792c69f902a7c5ca6 2023-03-20 10:57:56,943 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/026e32c9064f48e792c69f902a7c5ca6, entries=150, sequenceid=148, filesize=25.3 K 2023-03-20 10:57:56,944 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/B/6708edf3beec490ab0550473db651135 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/6708edf3beec490ab0550473db651135 2023-03-20 10:57:56,949 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/6708edf3beec490ab0550473db651135, entries=150, sequenceid=148, filesize=11.8 K 2023-03-20 10:57:56,950 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/C/a19007863ea047cba240f46626498f7b as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/a19007863ea047cba240f46626498f7b 2023-03-20 10:57:56,953 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/a19007863ea047cba240f46626498f7b, entries=150, sequenceid=148, filesize=11.8 K 2023-03-20 10:57:56,954 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for ee10ffd90d917f64b63db6b593dd0991 in 1362ms, sequenceid=148, compaction requested=false 2023-03-20 10:57:56,954 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ee10ffd90d917f64b63db6b593dd0991: 2023-03-20 10:57:56,954 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:56,954 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:56,954 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:56,954 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:56,954 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:56,955 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:56,955 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:56,955 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:56,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:56,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:56,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:56,956 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:56,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:56,956 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:56,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:56,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:56,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:56,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:56,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:56,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@533a70b3[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:56,957 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:56,957 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:56,958 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:56,958 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:56,959 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:56,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:56,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:56,959 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:56,959 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:56,959 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:56,960 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:56,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:56,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:56,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:56,960 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:56,960 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:56,961 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:56,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:56,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:56,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:56,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:56,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:56,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:56,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:56,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:56,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:56,964 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:56,964 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:56,964 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:56,964 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:56,964 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:56,969 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:56,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:56,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:56,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:56,970 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:56,970 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:56,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:56,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:56,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:56,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:56,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:56,971 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:56,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:56,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:56,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:56,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:56,971 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:56,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:56,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:56,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:56,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:56,980 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:56,980 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:56,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:56,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:56,981 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:56,981 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:56,981 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:56,981 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:56,981 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:56,981 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:56,981 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:56,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:56,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:56,982 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:56,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:56,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:56,982 DEBUG [Thread-2246] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:56,982 DEBUG [Thread-2246] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:57,083 DEBUG [Thread-2246] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:57,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:57,084 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:57,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:57,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:57,085 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:57,085 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:57,085 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:57,085 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:57,085 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:57,085 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:57,086 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:57,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:57,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:57,086 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:57,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:57,086 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:57,086 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:57,087 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:57,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:57,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:57,087 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:57,087 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:57,087 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:57,087 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:57,087 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:57,087 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:57,087 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:57,087 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:57,087 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:57,087 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. started... 2023-03-20 10:57:57,087 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ee10ffd90d917f64b63db6b593dd0991 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-20 10:57:57,088 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee10ffd90d917f64b63db6b593dd0991, store=A 2023-03-20 10:57:57,088 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:57,088 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee10ffd90d917f64b63db6b593dd0991, store=B 2023-03-20 10:57:57,088 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:57,088 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee10ffd90d917f64b63db6b593dd0991, store=C 2023-03-20 10:57:57,088 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:57,506 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230320763bac6158764697be62775ed97f9f0b to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320763bac6158764697be62775ed97f9f0b 2023-03-20 10:57:57,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on ee10ffd90d917f64b63db6b593dd0991 2023-03-20 10:57:57,861 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. as already flushing 2023-03-20 10:57:57,911 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:57,911 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=164, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/A/74f1892379b74f1580b12d64ef2aa019 2023-03-20 10:57:57,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43528 deadline: 1679309937906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:57,912 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:57,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43492 deadline: 1679309937907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:58,020 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:58,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43528 deadline: 1679309938013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:58,020 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:58,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43492 deadline: 1679309938014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:58,224 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:58,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43492 deadline: 1679309938221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:58,225 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:58,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43528 deadline: 1679309938222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:58,321 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=164 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/B/68a019c95427498c9c7777d7b863052c 2023-03-20 10:57:58,531 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:58,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43492 deadline: 1679309938525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:58,535 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:58,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43528 deadline: 1679309938528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:58,731 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=164 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/C/5c87af7b44664630b6c7848509329ea0 2023-03-20 10:57:58,735 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/A/74f1892379b74f1580b12d64ef2aa019 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/74f1892379b74f1580b12d64ef2aa019 2023-03-20 10:57:58,740 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/74f1892379b74f1580b12d64ef2aa019, entries=150, sequenceid=164, filesize=25.3 K 2023-03-20 10:57:58,741 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/B/68a019c95427498c9c7777d7b863052c as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/68a019c95427498c9c7777d7b863052c 2023-03-20 10:57:58,744 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/68a019c95427498c9c7777d7b863052c, entries=150, sequenceid=164, filesize=11.8 K 2023-03-20 10:57:58,744 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/C/5c87af7b44664630b6c7848509329ea0 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/5c87af7b44664630b6c7848509329ea0 2023-03-20 10:57:58,747 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/5c87af7b44664630b6c7848509329ea0, entries=150, sequenceid=164, filesize=11.8 K 2023-03-20 10:57:58,749 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for ee10ffd90d917f64b63db6b593dd0991 in 1662ms, sequenceid=164, compaction requested=true 2023-03-20 10:57:58,749 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ee10ffd90d917f64b63db6b593dd0991: 2023-03-20 10:57:58,749 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:58,749 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:58,749 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:58,749 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:58,749 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:58,749 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:58,755 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:58,755 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:58,755 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:58,756 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:58,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:58,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:58,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:58,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:58,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:58,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:58,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:58,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:58,767 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:58,767 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64d4a65f[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:58,767 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:58,767 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:58,770 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:58,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:58,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:58,770 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:58,770 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:58,770 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:58,770 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:58,770 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:58,771 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:58,771 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:58,771 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:58,771 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:58,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:58,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:58,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:58,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:58,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:58,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:58,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:58,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:58,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:58,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:58,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:58,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:58,773 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:58,773 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:58,773 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:58,773 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:58,773 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:58,774 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:58,774 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:58,775 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:58,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:58,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:58,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:58,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:58,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:58,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:58,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:58,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:58,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:58,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:58,776 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:58,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:58,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:58,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:58,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:58,777 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:58,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:58,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:58,778 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:58,778 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:58,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:58,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:58,779 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:58,779 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:58,779 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:58,779 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:58,779 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:58,779 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:58,779 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:58,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:58,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:58,779 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:58,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:58,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:58,779 DEBUG [Thread-2246] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:58,779 DEBUG [Thread-2246] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:58,879 DEBUG [Thread-2246] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:58,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:58,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:58,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:58,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:58,906 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:58,906 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:58,906 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:58,906 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:58,907 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:58,907 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:58,908 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:58,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:58,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:58,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:58,908 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:58,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:58,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:58,909 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:58,909 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:58,910 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:58,910 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:57:58,914 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:57:58,914 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:57:58,914 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:57:58,914 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:58,914 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:57:58,927 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:57:58,927 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:57:58,927 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:58,927 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. started... 2023-03-20 10:57:58,927 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ee10ffd90d917f64b63db6b593dd0991 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-20 10:57:58,928 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee10ffd90d917f64b63db6b593dd0991, store=A 2023-03-20 10:57:58,928 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:58,928 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee10ffd90d917f64b63db6b593dd0991, store=B 2023-03-20 10:57:58,928 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:58,928 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee10ffd90d917f64b63db6b593dd0991, store=C 2023-03-20 10:57:58,928 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:58,999 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303201f88a48d7e8b4964a484f3fcbc9df7f3 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303201f88a48d7e8b4964a484f3fcbc9df7f3 2023-03-20 10:57:59,039 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. as already flushing 2023-03-20 10:57:59,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on ee10ffd90d917f64b63db6b593dd0991 2023-03-20 10:57:59,096 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:59,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43492 deadline: 1679309939087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:59,097 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:59,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43528 deadline: 1679309939087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:59,205 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:59,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43492 deadline: 1679309939198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:59,206 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:59,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43528 deadline: 1679309939198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:59,408 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=182, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/A/c2b467af561c4214b43b705fe3d6e091 2023-03-20 10:57:59,416 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:59,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43528 deadline: 1679309939408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:59,417 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:59,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43492 deadline: 1679309939409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:59,432 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=182 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/B/7bf0ba77a62e46199b553fdf51c94dac 2023-03-20 10:57:59,721 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:59,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43492 deadline: 1679309939718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:59,721 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:59,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43528 deadline: 1679309939719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:59,847 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=182 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/C/d8c9d6bdad324d439706f0cd4e57d6b0 2023-03-20 10:57:59,851 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/A/c2b467af561c4214b43b705fe3d6e091 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/c2b467af561c4214b43b705fe3d6e091 2023-03-20 10:57:59,861 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/c2b467af561c4214b43b705fe3d6e091, entries=150, sequenceid=182, filesize=25.3 K 2023-03-20 10:57:59,862 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/B/7bf0ba77a62e46199b553fdf51c94dac as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/7bf0ba77a62e46199b553fdf51c94dac 2023-03-20 10:57:59,867 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/7bf0ba77a62e46199b553fdf51c94dac, entries=150, sequenceid=182, filesize=11.8 K 2023-03-20 10:57:59,868 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/C/d8c9d6bdad324d439706f0cd4e57d6b0 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/d8c9d6bdad324d439706f0cd4e57d6b0 2023-03-20 10:57:59,871 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/d8c9d6bdad324d439706f0cd4e57d6b0, entries=150, sequenceid=182, filesize=11.8 K 2023-03-20 10:57:59,872 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for ee10ffd90d917f64b63db6b593dd0991 in 945ms, sequenceid=182, compaction requested=true 2023-03-20 10:57:59,872 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ee10ffd90d917f64b63db6b593dd0991: 2023-03-20 10:57:59,872 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:57:59,872 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:57:59,872 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:57:59,872 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:59,872 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:57:59,872 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:57:59,873 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:59,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:59,873 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:59,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:59,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:59,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on ee10ffd90d917f64b63db6b593dd0991 2023-03-20 10:57:59,874 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ee10ffd90d917f64b63db6b593dd0991 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-20 10:57:59,874 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:59,874 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:57:59,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:59,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:59,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:59,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:59,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:59,875 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee10ffd90d917f64b63db6b593dd0991, store=A 2023-03-20 10:57:59,875 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:59,875 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee10ffd90d917f64b63db6b593dd0991, store=B 2023-03-20 10:57:59,876 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:59,876 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee10ffd90d917f64b63db6b593dd0991, store=C 2023-03-20 10:57:59,876 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:57:59,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:57:59,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23e2d2d5[Count = 0] remaining members to acquire global barrier 2023-03-20 10:57:59,876 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:57:59,876 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:59,877 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:59,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:59,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:59,877 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:57:59,877 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:57:59,877 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:57:59,877 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:59,877 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:57:59,878 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:59,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:59,878 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:57:59,878 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:59,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:59,878 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:57:59,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:59,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:59,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:59,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:59,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:59,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:59,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:59,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:59,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:57:59,883 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:57:59,883 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:57:59,883 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:57:59,883 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:57:59,883 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:57:59,884 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:57:59,885 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:59,885 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:59,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:59,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:57:59,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:57:59,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:59,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:59,885 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:59,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:59,885 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:59,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:57:59,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:59,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:59,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:59,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:57:59,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:59,886 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:59,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:59,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:57:59,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:57:59,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:59,888 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:59,888 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:59,888 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:59,888 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:59,888 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:59,888 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:57:59,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:59,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:59,888 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:57:59,888 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:59,888 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:57:59,889 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:59,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:57:59,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:57:59,888 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:57:59,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:57:59,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:57:59,889 DEBUG [Thread-2246] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:57:59,889 DEBUG [Thread-2246] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:57:59,906 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230320af5b92c660684fa28326d751f689a2c5 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320af5b92c660684fa28326d751f689a2c5 2023-03-20 10:57:59,928 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:59,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43494 deadline: 1679309939916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:59,930 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:59,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43478 deadline: 1679309939917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:59,930 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:57:59,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43530 deadline: 1679309939918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:59,989 DEBUG [Thread-2246] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:57:59,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:57:59,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:57:59,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:57:59,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:57:59,992 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:57:59,992 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:57:59,992 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:57:59,992 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:57:59,993 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:59,993 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:59,993 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:57:59,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:57:59,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:57:59,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:59,994 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:57:59,994 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:59,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:57:59,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:57:59,995 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:57:59,995 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:57:59,995 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:00,002 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:00,002 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:00,002 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:00,002 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:00,002 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:00,008 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:00,008 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:00,008 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:00,008 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. started... 2023-03-20 10:58:00,009 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. as already flushing 2023-03-20 10:58:00,009 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:00,009 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:00,009 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:00,009 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,009 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:00,009 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:00,010 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,010 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:00,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:00,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:00,010 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:00,010 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:00,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:00,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:00,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:00,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:00,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:00,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5629a9ee[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:00,011 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:00,012 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:00,012 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:00,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:00,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:00,013 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:00,013 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:00,013 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:00,014 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,014 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:00,014 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:00,014 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:00,014 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,014 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:00,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:00,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:00,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:00,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:00,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:00,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:00,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:00,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:00,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:00,016 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:00,016 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:00,017 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:00,017 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:00,017 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:00,018 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:00,018 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:00,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:00,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:00,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:00,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:00,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:00,018 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:00,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:00,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:00,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:00,019 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:00,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:00,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:00,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:00,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,021 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:00,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:00,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,029 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:00,029 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,029 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:00,029 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:00,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:00,029 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:00,029 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:00,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:00,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:00,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:00,029 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,030 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:00,030 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:00,030 DEBUG [Thread-2246] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:00,030 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:00,030 DEBUG [Thread-2246] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:00,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:00,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:00,039 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:00,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43478 deadline: 1679309940031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,039 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:00,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43494 deadline: 1679309940032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,040 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:00,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43530 deadline: 1679309940035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,130 DEBUG [Thread-2246] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:00,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:00,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:00,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:00,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:00,134 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:00,134 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:00,134 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:00,134 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:00,135 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:00,135 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:00,135 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,135 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:00,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:00,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:00,138 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,138 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:00,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:00,138 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:00,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:00,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:00,139 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:00,140 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:00,140 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:00,140 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:00,140 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:00,140 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:00,141 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:00,141 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:00,141 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:00,141 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. started... 2023-03-20 10:58:00,141 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. as already flushing 2023-03-20 10:58:00,141 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:00,141 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:00,141 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:00,141 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,142 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:00,142 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:00,145 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,146 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:00,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:00,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:00,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:00,146 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:00,146 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:00,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:00,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:00,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:00,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:00,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5616e470[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:00,148 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:00,148 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:00,148 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:00,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:00,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:00,148 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,148 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:00,148 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:00,149 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:00,149 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:00,149 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,149 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:00,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:00,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:00,149 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:00,149 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:00,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:00,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:00,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:00,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:00,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:00,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:00,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:00,151 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:00,151 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:00,151 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:00,151 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:00,151 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:00,153 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:00,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:00,153 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:00,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:00,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:00,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:00,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:00,154 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,154 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,156 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,156 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:00,156 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:00,156 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,156 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:00,156 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:00,156 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:00,156 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:00,156 DEBUG [Thread-2246] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:00,156 DEBUG [Thread-2246] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:00,153 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:00,156 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:00,156 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:00,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:00,157 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:00,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:00,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:00,158 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:00,158 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:00,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:00,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:00,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:00,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:00,232 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:00,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43528 deadline: 1679309940223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,233 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:00,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43492 deadline: 1679309940223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,245 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:00,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43494 deadline: 1679309940241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,245 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:00,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43478 deadline: 1679309940241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,245 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:00,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43530 deadline: 1679309940242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,256 DEBUG [Thread-2246] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:00,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:00,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:00,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:00,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:00,259 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:00,259 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:00,259 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:00,259 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:00,260 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:00,260 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:00,261 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:00,261 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:00,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:00,261 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,261 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:00,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:00,262 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:00,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:00,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:00,262 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:00,262 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:00,262 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:00,262 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:00,262 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:00,262 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:00,269 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:00,269 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:00,269 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:00,269 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. started... 2023-03-20 10:58:00,269 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. as already flushing 2023-03-20 10:58:00,269 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:00,269 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:00,269 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:00,269 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,269 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:00,269 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:00,270 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:00,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:00,270 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:00,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:00,270 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:00,270 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:00,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:00,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:00,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:00,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:00,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17b156b3[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:00,271 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:00,271 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:00,272 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:00,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:00,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:00,272 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:00,272 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:00,272 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:00,272 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,272 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:00,273 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:00,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:00,273 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:00,273 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:00,273 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:00,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:00,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:00,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:00,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:00,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:00,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:00,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:00,275 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:00,275 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:00,275 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:00,275 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:00,275 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:00,275 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:00,275 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:00,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:00,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:00,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:00,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:00,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:00,276 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:00,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:00,276 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:00,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:00,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:00,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:00,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:00,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:00,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,277 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:00,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:00,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,279 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:00,279 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:00,279 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:00,279 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:00,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:00,279 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:00,279 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:00,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:00,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:00,280 DEBUG [Thread-2246] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:00,280 DEBUG [Thread-2246] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:00,280 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:00,280 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,280 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:00,280 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:00,280 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:00,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:00,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:00,344 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=198, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/A/88d408d50c91415ba1c4db6cdfc749c1 2023-03-20 10:58:00,380 DEBUG [Thread-2246] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:00,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:00,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:00,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:00,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:00,385 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:00,385 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:00,385 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:00,385 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:00,386 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:00,386 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:00,387 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:00,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:00,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:00,387 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:00,387 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,388 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:00,388 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:00,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:00,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:00,388 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:00,388 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:00,388 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:00,388 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:00,388 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:00,388 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:00,390 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:00,390 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:00,390 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:00,390 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. started... 2023-03-20 10:58:00,390 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. as already flushing 2023-03-20 10:58:00,390 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:00,390 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:00,390 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:00,390 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,390 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:00,390 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:00,394 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:00,395 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:00,395 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,395 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:00,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:00,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:00,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:00,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:00,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:00,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:00,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:00,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22705aea[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:00,396 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:00,396 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:00,397 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:00,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:00,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:00,397 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:00,397 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:00,397 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:00,397 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,397 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:00,398 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,398 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:00,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:00,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:00,398 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:00,398 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:00,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:00,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:00,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:00,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:00,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:00,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:00,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:00,402 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:00,402 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:00,402 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:00,402 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:00,402 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:00,411 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:00,411 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:00,411 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:00,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:00,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:00,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:00,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:00,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:00,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:00,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:00,412 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:00,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:00,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:00,413 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:00,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:00,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:00,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:00,416 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:00,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:00,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:00,416 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:00,416 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:00,416 DEBUG [Thread-2246] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:00,416 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:00,416 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,417 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:00,417 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:00,417 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,417 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:00,418 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:00,418 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:00,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:00,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:00,416 DEBUG [Thread-2246] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:00,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:00,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:00,518 DEBUG [Thread-2246] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:00,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:00,529 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:00,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:00,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:00,542 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:00,542 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:00,542 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:00,542 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:00,544 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:00,544 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:00,545 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:00,545 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:00,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:00,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:00,546 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,546 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:00,546 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:00,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:00,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:00,546 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:00,550 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:00,550 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:00,551 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:00,551 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:00,551 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:00,554 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:00,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43478 deadline: 1679309940547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,555 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:00,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43494 deadline: 1679309940547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,555 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:00,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43530 deadline: 1679309940547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,558 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:00,558 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:00,558 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:00,558 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. started... 2023-03-20 10:58:00,558 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. as already flushing 2023-03-20 10:58:00,558 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:00,562 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:00,562 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:00,562 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,562 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:00,562 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:00,563 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:00,563 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:00,563 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:00,566 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:00,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:00,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:00,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:00,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:00,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:00,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:00,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@398df193[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:00,568 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:00,568 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:00,569 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:00,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:00,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:00,570 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,570 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:00,570 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:00,570 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:00,570 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:00,574 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:00,574 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,574 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:00,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:00,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:00,574 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:00,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:00,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:00,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:00,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:00,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:00,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:00,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:00,588 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:00,588 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:00,588 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:00,588 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:00,588 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:00,589 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:00,590 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:00,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:00,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:00,590 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:00,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:00,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:00,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:00,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:00,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:00,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:00,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:00,593 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:00,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:00,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:00,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,594 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:00,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:00,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,597 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:00,597 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:00,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:00,597 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:00,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:00,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:00,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:00,597 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,597 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:00,597 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:00,597 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:00,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:00,597 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:00,597 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:00,597 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:00,597 DEBUG [Thread-2246] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:00,597 DEBUG [Thread-2246] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:00,697 DEBUG [Thread-2246] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:00,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:00,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:00,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:00,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:00,702 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:00,702 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:00,702 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:00,702 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:00,710 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:00,710 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:00,711 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,711 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,711 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:00,714 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:00,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:00,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:00,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:00,720 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:00,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:00,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:00,720 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:00,726 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:00,726 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:00,726 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:00,726 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:00,726 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:00,735 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:00,735 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:00,735 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:00,735 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. started... 2023-03-20 10:58:00,735 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. as already flushing 2023-03-20 10:58:00,735 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:00,735 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:00,735 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:00,735 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,735 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:00,735 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:00,737 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:00,737 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,737 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:00,737 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:00,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:00,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:00,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:00,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:00,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:00,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:00,740 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:00,740 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5859a7b6[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:00,740 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:00,740 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:00,741 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:00,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:00,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:00,742 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:00,742 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:00,742 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:00,743 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:00,743 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:00,743 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:00,744 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,744 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,744 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:00,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:00,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:00,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:00,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:00,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:00,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:00,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:00,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:00,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:00,752 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:00,752 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:00,752 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:00,752 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:00,752 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:00,761 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=198 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/B/c44c44475b51498b9cac38c24c7b1840 2023-03-20 10:58:00,761 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:00,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:00,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:00,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:00,762 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:00,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:00,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:00,764 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:00,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:00,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:00,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,770 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,770 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:00,770 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:00,770 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,770 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:00,770 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:00,770 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:00,770 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:00,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:00,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:00,771 DEBUG [Thread-2246] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:00,771 DEBUG [Thread-2246] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:00,778 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:00,778 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:00,778 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:00,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:00,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:00,790 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:00,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:00,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:00,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:00,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:00,815 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=198 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/C/643397ed8be644b7b4c68ba9a8328a87 2023-03-20 10:58:00,831 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/A/88d408d50c91415ba1c4db6cdfc749c1 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/88d408d50c91415ba1c4db6cdfc749c1 2023-03-20 10:58:00,839 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/88d408d50c91415ba1c4db6cdfc749c1, entries=250, sequenceid=198, filesize=38.9 K 2023-03-20 10:58:00,840 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/B/c44c44475b51498b9cac38c24c7b1840 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/c44c44475b51498b9cac38c24c7b1840 2023-03-20 10:58:00,844 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/c44c44475b51498b9cac38c24c7b1840, entries=150, sequenceid=198, filesize=11.8 K 2023-03-20 10:58:00,850 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/C/643397ed8be644b7b4c68ba9a8328a87 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/643397ed8be644b7b4c68ba9a8328a87 2023-03-20 10:58:00,858 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/643397ed8be644b7b4c68ba9a8328a87, entries=150, sequenceid=198, filesize=11.8 K 2023-03-20 10:58:00,859 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for ee10ffd90d917f64b63db6b593dd0991 in 985ms, sequenceid=198, compaction requested=true 2023-03-20 10:58:00,859 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ee10ffd90d917f64b63db6b593dd0991: 2023-03-20 10:58:00,859 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:58:00,859 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:58:00,859 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-20 10:58:00,860 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-20 10:58:00,860 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-20 10:58:00,862 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 143638 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-20 10:58:00,863 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): ee10ffd90d917f64b63db6b593dd0991/A is initiating minor compaction (all files) 2023-03-20 10:58:00,863 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ee10ffd90d917f64b63db6b593dd0991/A in TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:00,863 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/d496e90bb995417c9b71d15bafb8fc99, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/026e32c9064f48e792c69f902a7c5ca6, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/74f1892379b74f1580b12d64ef2aa019, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/c2b467af561c4214b43b705fe3d6e091, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/88d408d50c91415ba1c4db6cdfc749c1] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp, totalSize=140.3 K 2023-03-20 10:58:00,863 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting d496e90bb995417c9b71d15bafb8fc99, keycount=150, bloomtype=ROW, size=25.5 K, encoding=NONE, compression=NONE, seqNum=129, earliestPutTs=1679309874594 2023-03-20 10:58:00,864 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 026e32c9064f48e792c69f902a7c5ca6, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1679309875041 2023-03-20 10:58:00,864 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 74f1892379b74f1580b12d64ef2aa019, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=164, earliestPutTs=1679309875687 2023-03-20 10:58:00,865 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting c2b467af561c4214b43b705fe3d6e091, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1679309877895 2023-03-20 10:58:00,867 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 88d408d50c91415ba1c4db6cdfc749c1, keycount=250, bloomtype=ROW, size=38.9 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1679309879038 2023-03-20 10:58:00,871 DEBUG [Thread-2246] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:00,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:00,874 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60354 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-20 10:58:00,874 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1912): ee10ffd90d917f64b63db6b593dd0991/B is initiating minor compaction (all files) 2023-03-20 10:58:00,874 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ee10ffd90d917f64b63db6b593dd0991/B in TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:00,874 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/d81acef791404fc09c434ff0ea87e8f0, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/6708edf3beec490ab0550473db651135, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/68a019c95427498c9c7777d7b863052c, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/7bf0ba77a62e46199b553fdf51c94dac, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/c44c44475b51498b9cac38c24c7b1840] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp, totalSize=58.9 K 2023-03-20 10:58:00,874 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:00,875 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting d81acef791404fc09c434ff0ea87e8f0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=129, earliestPutTs=1679309874594 2023-03-20 10:58:00,876 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 6708edf3beec490ab0550473db651135, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1679309875041 2023-03-20 10:58:00,877 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 68a019c95427498c9c7777d7b863052c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=164, earliestPutTs=1679309875687 2023-03-20 10:58:00,878 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 7bf0ba77a62e46199b553fdf51c94dac, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1679309877895 2023-03-20 10:58:00,878 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting c44c44475b51498b9cac38c24c7b1840, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1679309879038 2023-03-20 10:58:00,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:00,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:00,886 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:00,886 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:00,886 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:00,886 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:00,888 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:00,888 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:00,888 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:00,888 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:00,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:00,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:00,889 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:00,889 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:00,889 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:00,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:00,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:00,889 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:00,894 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:00,894 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:00,894 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:00,894 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:00,894 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:00,900 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] throttle.PressureAwareThroughputController(145): ee10ffd90d917f64b63db6b593dd0991#B#compaction#337 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:58:00,904 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): ee10ffd90d917f64b63db6b593dd0991#A#compaction#336 average throughput is 1.41 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:58:00,906 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:00,906 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:00,906 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:00,906 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. started... 2023-03-20 10:58:00,906 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ee10ffd90d917f64b63db6b593dd0991 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-20 10:58:00,906 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee10ffd90d917f64b63db6b593dd0991, store=A 2023-03-20 10:58:00,906 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:00,906 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee10ffd90d917f64b63db6b593dd0991, store=B 2023-03-20 10:58:00,907 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:00,907 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee10ffd90d917f64b63db6b593dd0991, store=C 2023-03-20 10:58:00,907 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:00,927 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/B/90458d0d168c42e6be839aa6ceeb1bc3 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/90458d0d168c42e6be839aa6ceeb1bc3 2023-03-20 10:58:00,941 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in ee10ffd90d917f64b63db6b593dd0991/B of ee10ffd90d917f64b63db6b593dd0991 into 90458d0d168c42e6be839aa6ceeb1bc3(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:58:00,941 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ee10ffd90d917f64b63db6b593dd0991: 2023-03-20 10:58:00,941 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991., storeName=ee10ffd90d917f64b63db6b593dd0991/B, priority=11, startTime=1679309880859; duration=0sec 2023-03-20 10:58:00,941 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:58:00,941 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-20 10:58:00,944 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60354 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-20 10:58:00,944 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1912): ee10ffd90d917f64b63db6b593dd0991/C is initiating minor compaction (all files) 2023-03-20 10:58:00,944 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ee10ffd90d917f64b63db6b593dd0991/C in TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:00,944 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/08c958c21ecb4751a10a92ffdbe1feaa, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/a19007863ea047cba240f46626498f7b, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/5c87af7b44664630b6c7848509329ea0, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/d8c9d6bdad324d439706f0cd4e57d6b0, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/643397ed8be644b7b4c68ba9a8328a87] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp, totalSize=58.9 K 2023-03-20 10:58:00,946 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 08c958c21ecb4751a10a92ffdbe1feaa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=129, earliestPutTs=1679309874594 2023-03-20 10:58:00,946 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting a19007863ea047cba240f46626498f7b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1679309875041 2023-03-20 10:58:00,947 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 5c87af7b44664630b6c7848509329ea0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=164, earliestPutTs=1679309875687 2023-03-20 10:58:00,947 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting d8c9d6bdad324d439706f0cd4e57d6b0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1679309877895 2023-03-20 10:58:00,947 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 643397ed8be644b7b4c68ba9a8328a87, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1679309879038 2023-03-20 10:58:00,949 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032057a584a8ac2a485fb356e72a271f3b8b to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032057a584a8ac2a485fb356e72a271f3b8b 2023-03-20 10:58:00,955 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] throttle.PressureAwareThroughputController(145): ee10ffd90d917f64b63db6b593dd0991#C#compaction#339 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:58:00,990 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=214, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/A/26340732b60145d1af43001b00da6166 2023-03-20 10:58:01,009 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/C/3ed679e38c8c47d79b39dd77b5989c68 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/3ed679e38c8c47d79b39dd77b5989c68 2023-03-20 10:58:01,014 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in ee10ffd90d917f64b63db6b593dd0991/C of ee10ffd90d917f64b63db6b593dd0991 into 3ed679e38c8c47d79b39dd77b5989c68(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:58:01,014 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ee10ffd90d917f64b63db6b593dd0991: 2023-03-20 10:58:01,014 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991., storeName=ee10ffd90d917f64b63db6b593dd0991/C, priority=11, startTime=1679309880860; duration=0sec 2023-03-20 10:58:01,014 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:58:01,028 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=214 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/B/282030eb96ea423e8a055a9b262155b0 2023-03-20 10:58:01,056 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=214 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/C/1745645435e14d108c602095c57ca9cf 2023-03-20 10:58:01,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on ee10ffd90d917f64b63db6b593dd0991 2023-03-20 10:58:01,061 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. as already flushing 2023-03-20 10:58:01,064 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/A/26340732b60145d1af43001b00da6166 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/26340732b60145d1af43001b00da6166 2023-03-20 10:58:01,071 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/26340732b60145d1af43001b00da6166, entries=150, sequenceid=214, filesize=25.3 K 2023-03-20 10:58:01,074 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/B/282030eb96ea423e8a055a9b262155b0 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/282030eb96ea423e8a055a9b262155b0 2023-03-20 10:58:01,086 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/282030eb96ea423e8a055a9b262155b0, entries=150, sequenceid=214, filesize=11.8 K 2023-03-20 10:58:01,098 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/C/1745645435e14d108c602095c57ca9cf as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/1745645435e14d108c602095c57ca9cf 2023-03-20 10:58:01,108 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:01,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43494 deadline: 1679309941100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,109 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:01,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43530 deadline: 1679309941100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,109 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:01,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43478 deadline: 1679309941101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,118 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/1745645435e14d108c602095c57ca9cf, entries=150, sequenceid=214, filesize=11.8 K 2023-03-20 10:58:01,119 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for ee10ffd90d917f64b63db6b593dd0991 in 213ms, sequenceid=214, compaction requested=false 2023-03-20 10:58:01,120 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ee10ffd90d917f64b63db6b593dd0991: 2023-03-20 10:58:01,120 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:01,120 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:01,120 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:01,120 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,122 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:01,122 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:01,125 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:01,125 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:01,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:01,125 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:01,126 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:01,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:01,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:01,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:01,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:01,127 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:01,127 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:01,127 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:01,127 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45c076c0[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:01,128 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:01,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:01,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:01,128 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:01,128 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:01,128 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:01,128 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,128 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:01,129 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:01,129 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:01,129 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:01,129 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:01,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:01,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:01,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:01,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:01,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:01,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:01,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:01,132 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:01,132 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:01,132 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:01,132 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:01,132 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:01,132 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:01,133 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:01,133 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:01,133 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:01,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:01,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:01,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:01,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:01,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:01,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:01,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:01,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:01,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:01,135 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:01,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:01,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:01,136 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:01,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:01,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,145 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:01,145 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:01,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:01,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:01,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:01,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:01,146 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,146 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:01,146 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:01,146 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:01,146 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:01,146 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:01,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:01,146 DEBUG [Thread-2246] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:01,146 DEBUG [Thread-2246] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:01,146 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:01,147 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:01,147 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:01,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on ee10ffd90d917f64b63db6b593dd0991 2023-03-20 10:58:01,214 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ee10ffd90d917f64b63db6b593dd0991 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-20 10:58:01,214 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee10ffd90d917f64b63db6b593dd0991, store=A 2023-03-20 10:58:01,214 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:01,214 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee10ffd90d917f64b63db6b593dd0991, store=B 2023-03-20 10:58:01,214 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:01,214 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee10ffd90d917f64b63db6b593dd0991, store=C 2023-03-20 10:58:01,214 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:01,246 DEBUG [Thread-2246] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:01,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:01,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:01,253 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230320d7b01b2d63f644d9bb51dffb3eece691 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320d7b01b2d63f644d9bb51dffb3eece691 2023-03-20 10:58:01,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:01,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:01,255 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:01,255 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:01,255 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:01,255 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:01,256 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:01,256 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:01,256 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:01,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43530 deadline: 1679309941242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,257 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:01,257 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:01,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:01,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:01,258 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,258 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:01,258 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:01,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:01,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:01,258 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:01,259 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:01,259 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:01,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43494 deadline: 1679309941248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,259 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:01,259 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:01,259 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:01,259 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:01,259 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:01,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43492 deadline: 1679309941249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,259 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:01,259 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:01,259 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:01,259 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. started... 2023-03-20 10:58:01,259 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. as already flushing 2023-03-20 10:58:01,259 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:01,259 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:01,260 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:01,260 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,260 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:01,260 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:01,261 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:01,261 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:01,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:01,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:01,261 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:01,261 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:01,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:01,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:01,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:01,262 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:01,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:01,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43478 deadline: 1679309941255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6547d398[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:01,263 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:01,263 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:01,263 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:01,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:01,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:01,263 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:01,264 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:01,264 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:01,264 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,264 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:01,265 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:01,265 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:01,265 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,265 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:01,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:01,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:01,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:01,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:01,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:01,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:01,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:01,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:01,268 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:01,268 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:01,268 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:01,268 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:01,268 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:01,268 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:01,269 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:01,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43528 deadline: 1679309941257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,269 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:01,269 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:01,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:01,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:01,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:01,269 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:01,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:01,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:01,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:01,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:01,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:01,271 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:01,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:01,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:01,272 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:01,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:01,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:01,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,275 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:01,275 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:01,275 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,275 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:01,275 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:01,275 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,275 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:01,275 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:01,275 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:01,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:01,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:01,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:01,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:01,275 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:01,275 DEBUG [Thread-2246] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:01,275 DEBUG [Thread-2246] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:01,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:01,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:01,276 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:01,367 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:01,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43530 deadline: 1679309941357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,368 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:01,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43478 deadline: 1679309941367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,368 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:01,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43494 deadline: 1679309941367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,376 DEBUG [Thread-2246] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:01,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:01,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:01,380 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:01,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43492 deadline: 1679309941370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,381 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:01,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43528 deadline: 1679309941370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:01,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:01,394 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:01,394 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:01,394 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:01,394 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:01,399 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:01,399 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:01,400 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:01,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:01,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:01,400 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:01,402 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:01,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:01,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:01,403 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:01,404 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,404 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:01,404 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:01,404 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:01,404 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:01,404 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:01,404 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:01,406 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:01,406 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:01,410 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:01,410 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. started... 2023-03-20 10:58:01,410 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. as already flushing 2023-03-20 10:58:01,410 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:01,410 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:01,410 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:01,410 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,410 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:01,410 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:01,413 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,413 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:01,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:01,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:01,413 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:01,413 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:01,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:01,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:01,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:01,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:01,415 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:01,415 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3310b7ca[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:01,415 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:01,415 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:01,416 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:01,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:01,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:01,416 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:01,416 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:01,416 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:01,416 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,416 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:01,416 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:01,417 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:01,417 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:01,416 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:01,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:01,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:01,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:01,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:01,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:01,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:01,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:01,419 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:01,419 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:01,419 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:01,419 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:01,419 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:01,419 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:01,420 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:01,420 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:01,420 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:01,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:01,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:01,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:01,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:01,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:01,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:01,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:01,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:01,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:01,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:01,421 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:01,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:01,430 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:01,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:01,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,436 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:01,436 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:01,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:01,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:01,437 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,437 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:01,437 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:01,437 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,437 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:01,437 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:01,437 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:01,437 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:01,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:01,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:01,437 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:01,437 DEBUG [Thread-2246] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:01,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:01,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:01,437 DEBUG [Thread-2246] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:01,538 DEBUG [Thread-2246] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:01,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:01,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:01,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:01,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:01,542 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:01,542 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:01,542 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:01,542 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:01,543 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:01,543 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:01,545 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:01,545 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:01,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:01,545 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,545 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:01,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:01,546 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:01,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:01,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:01,548 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:01,557 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:01,557 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:01,557 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:01,558 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:01,558 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:01,558 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:01,558 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:01,558 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:01,558 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. started... 2023-03-20 10:58:01,558 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. as already flushing 2023-03-20 10:58:01,558 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:01,558 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:01,558 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:01,558 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,558 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:01,558 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:01,561 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,561 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:01,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:01,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:01,561 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:01,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:01,561 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:01,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:01,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:01,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:01,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:01,562 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:01,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2661c209[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:01,562 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:01,563 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:01,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:01,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:01,563 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:01,563 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:01,563 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:01,564 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:01,564 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:01,564 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:01,564 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:01,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:01,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:01,565 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,565 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:01,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:01,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:01,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:01,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:01,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:01,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:01,567 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:01,567 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:01,567 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:01,567 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:01,567 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:01,569 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:01,569 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:01,569 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:01,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:01,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:01,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:01,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:01,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:01,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:01,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:01,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:01,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:01,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:01,571 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:01,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:01,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,571 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:01,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:01,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,573 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,573 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:01,573 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:01,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:01,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:01,573 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:01,573 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:01,573 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,573 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:01,573 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:01,573 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:01,573 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:01,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:01,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:01,573 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:01,573 DEBUG [Thread-2246] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:01,573 DEBUG [Thread-2246] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:01,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43478 deadline: 1679309941569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,574 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:01,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43530 deadline: 1679309941569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:01,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:01,578 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:01,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43494 deadline: 1679309941570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,592 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:01,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43492 deadline: 1679309941583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,593 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:01,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43528 deadline: 1679309941583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,661 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=233, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/A/98228da5c6584de4b6fffc4e081c9fbf 2023-03-20 10:58:01,674 DEBUG [Thread-2246] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:01,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:01,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:01,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:01,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:01,682 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:01,682 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:01,682 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:01,682 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:01,683 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:01,683 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:01,683 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:01,683 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:01,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:01,684 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,684 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:01,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:01,684 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:01,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:01,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:01,684 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:01,690 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:01,690 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:01,690 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:01,690 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:01,690 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:01,694 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:01,694 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:01,694 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:01,694 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. started... 2023-03-20 10:58:01,694 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. as already flushing 2023-03-20 10:58:01,694 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:01,694 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:01,694 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:01,694 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,694 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:01,694 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:01,695 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,695 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:01,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:01,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:01,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:01,696 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:01,696 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:01,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:01,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:01,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:01,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:01,697 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:01,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6051c673[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:01,697 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:01,697 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:01,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:01,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:01,698 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:01,698 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:01,698 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:01,698 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,698 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:01,699 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:01,699 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:01,699 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:01,699 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:01,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:01,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:01,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:01,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:01,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:01,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:01,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:01,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:01,701 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:01,701 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:01,701 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:01,701 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:01,701 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:01,702 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:01,702 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:01,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:01,703 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:01,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:01,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:01,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:01,704 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:01,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:01,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:01,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:01,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:01,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:01,708 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:01,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:01,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:01,712 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,712 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:01,712 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:01,712 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:01,712 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:01,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:01,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:01,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:01,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:01,712 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:01,712 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,713 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:01,713 DEBUG [Thread-2246] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:01,713 DEBUG [Thread-2246] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:01,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:01,713 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:01,713 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:01,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:01,713 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:01,813 DEBUG [Thread-2246] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:01,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:01,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:01,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:01,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:01,816 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:01,816 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:01,816 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:01,816 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:01,816 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:01,816 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:01,817 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,817 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,817 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:01,818 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:01,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:01,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:01,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:01,818 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:01,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:01,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:01,819 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:01,822 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:01,822 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:01,822 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:01,822 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:01,822 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:01,830 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:01,830 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:01,830 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:01,830 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. started... 2023-03-20 10:58:01,830 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. as already flushing 2023-03-20 10:58:01,830 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:01,830 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:01,831 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:01,831 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,831 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:01,831 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:01,832 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:01,832 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:01,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:01,832 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:01,832 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:01,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:01,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:01,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:01,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:01,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:01,833 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:01,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3bd102f2[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:01,833 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:01,834 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:01,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:01,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:01,834 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:01,834 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:01,834 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:01,834 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,834 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:01,835 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:01,835 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:01,835 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:01,840 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:01,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:01,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:01,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:01,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:01,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:01,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:01,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:01,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:01,844 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:01,844 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:01,844 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:01,844 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:01,844 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:01,845 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:01,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:01,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:01,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:01,846 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,847 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:01,847 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:01,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:01,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:01,847 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:01,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:01,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:01,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:01,850 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:01,850 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:01,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:01,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:01,850 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,850 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:01,850 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:01,851 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,851 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:01,851 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:01,851 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:01,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:01,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:01,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:01,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:01,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:01,851 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:01,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:01,851 DEBUG [Thread-2246] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:01,851 DEBUG [Thread-2246] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:01,851 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:01,885 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:01,885 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:01,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43530 deadline: 1679309941876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43478 deadline: 1679309941876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,886 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:01,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43494 deadline: 1679309941881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,900 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:01,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43528 deadline: 1679309941895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,901 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:01,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43492 deadline: 1679309941896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,951 DEBUG [Thread-2246] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:01,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:01,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:01,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:01,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:01,962 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:01,962 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:01,962 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:01,962 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:01,964 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:01,964 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:01,964 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:01,964 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:01,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:01,965 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,965 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:01,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:01,965 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:01,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:01,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:01,965 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:01,965 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:01,966 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:01,966 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:01,966 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:01,966 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:01,970 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:01,970 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:01,971 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:01,971 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. started... 2023-03-20 10:58:01,971 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. as already flushing 2023-03-20 10:58:01,971 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:01,972 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:01,972 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:01,972 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,972 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:01,972 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:01,978 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:01,978 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:01,978 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:01,982 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:01,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:01,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:01,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:01,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:01,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:01,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:01,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@dc94ca5[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:01,985 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:01,985 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:01,986 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:01,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:01,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:01,987 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:01,987 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:01,987 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,987 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:01,987 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:01,988 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:01,988 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,988 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:01,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:01,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:01,988 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:01,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:01,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:01,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:01,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:01,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:01,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:01,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:01,990 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:01,990 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:01,990 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:01,990 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:01,990 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:01,991 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:01,991 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:01,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:01,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:01,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:01,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:01,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:01,993 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:01,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:01,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:01,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:01,994 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:01,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:01,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:01,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:01,996 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:01,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:01,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:01,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:02,003 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:02,003 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:02,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:02,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:02,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:02,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:02,004 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:02,004 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:02,004 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:02,004 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:02,004 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:02,004 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:02,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:02,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:02,004 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:02,005 DEBUG [Thread-2246] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:02,005 DEBUG [Thread-2246] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:02,004 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:02,005 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:02,094 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=233 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/B/a084c7d4e8464f1a8f5feaa7d12465b4 2023-03-20 10:58:02,105 DEBUG [Thread-2246] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:02,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:02,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:02,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:02,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:02,118 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:02,118 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:02,118 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:02,118 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:02,119 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:02,119 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:02,119 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:02,120 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:02,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:02,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:02,120 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:02,120 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:02,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:02,121 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:02,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:02,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:02,121 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:02,134 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:02,134 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:02,134 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:02,134 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:02,134 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:02,147 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:02,147 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:02,147 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:02,147 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=233 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/C/c0f6ef5209844c2c9a8dab31926d3764 2023-03-20 10:58:02,147 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. started... 2023-03-20 10:58:02,147 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. as already flushing 2023-03-20 10:58:02,147 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:02,147 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:02,147 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:02,147 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:02,147 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:02,147 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:02,148 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:02,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:02,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:02,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:02,148 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:02,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:02,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:02,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:02,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:02,151 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:02,151 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:02,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:02,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:02,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2590a69f[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:02,152 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:02,152 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:02,154 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:02,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:02,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:02,155 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:02,155 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:02,155 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:02,155 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:02,155 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:02,156 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/A/98228da5c6584de4b6fffc4e081c9fbf as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/98228da5c6584de4b6fffc4e081c9fbf 2023-03-20 10:58:02,156 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:02,156 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:02,156 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:02,156 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:02,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:02,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:02,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:02,160 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/98228da5c6584de4b6fffc4e081c9fbf, entries=200, sequenceid=233, filesize=32.1 K 2023-03-20 10:58:02,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:02,161 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/B/a084c7d4e8464f1a8f5feaa7d12465b4 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/a084c7d4e8464f1a8f5feaa7d12465b4 2023-03-20 10:58:02,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:02,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:02,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:02,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:02,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:02,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:02,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:02,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:02,165 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:02,165 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:02,165 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:02,165 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:02,165 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:02,166 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:02,166 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:02,166 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:02,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:02,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:02,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:02,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:02,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:02,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:02,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:02,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:02,167 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:02,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:02,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:02,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:02,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:02,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:02,168 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:02,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:02,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:02,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:02,172 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/a084c7d4e8464f1a8f5feaa7d12465b4, entries=150, sequenceid=233, filesize=11.8 K 2023-03-20 10:58:02,173 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/C/c0f6ef5209844c2c9a8dab31926d3764 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/c0f6ef5209844c2c9a8dab31926d3764 2023-03-20 10:58:02,175 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:02,175 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:02,175 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:02,175 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:02,175 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:02,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:02,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:02,175 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:02,175 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:02,175 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:02,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:02,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:02,175 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:02,175 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:02,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:02,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:02,176 DEBUG [Thread-2246] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:02,176 DEBUG [Thread-2246] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:02,183 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/c0f6ef5209844c2c9a8dab31926d3764, entries=150, sequenceid=233, filesize=11.8 K 2023-03-20 10:58:02,184 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for ee10ffd90d917f64b63db6b593dd0991 in 971ms, sequenceid=233, compaction requested=true 2023-03-20 10:58:02,184 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ee10ffd90d917f64b63db6b593dd0991: 2023-03-20 10:58:02,184 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:58:02,184 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:58:02,184 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 5 compacting, 2 eligible, 16 blocking 2023-03-20 10:58:02,184 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-20 10:58:02,187 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-20 10:58:02,188 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-20 10:58:02,188 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. because compaction request was cancelled 2023-03-20 10:58:02,188 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 10:58:02,188 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36552 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 10:58:02,206 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1912): ee10ffd90d917f64b63db6b593dd0991/B is initiating minor compaction (all files) 2023-03-20 10:58:02,207 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ee10ffd90d917f64b63db6b593dd0991/B in TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:02,207 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/90458d0d168c42e6be839aa6ceeb1bc3, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/282030eb96ea423e8a055a9b262155b0, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/a084c7d4e8464f1a8f5feaa7d12465b4] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp, totalSize=35.7 K 2023-03-20 10:58:02,207 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 90458d0d168c42e6be839aa6ceeb1bc3, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1679309879038 2023-03-20 10:58:02,208 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 282030eb96ea423e8a055a9b262155b0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=214, earliestPutTs=1679309879906 2023-03-20 10:58:02,208 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting a084c7d4e8464f1a8f5feaa7d12465b4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=233, earliestPutTs=1679309881088 2023-03-20 10:58:02,216 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/A/45da4837dcb74f89abc1e30f6855f2a0 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/45da4837dcb74f89abc1e30f6855f2a0 2023-03-20 10:58:02,219 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] throttle.PressureAwareThroughputController(145): ee10ffd90d917f64b63db6b593dd0991#B#compaction#345 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:58:02,224 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in ee10ffd90d917f64b63db6b593dd0991/A of ee10ffd90d917f64b63db6b593dd0991 into 45da4837dcb74f89abc1e30f6855f2a0(size=25.7 K), total size for store is 83.1 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-03-20 10:58:02,224 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ee10ffd90d917f64b63db6b593dd0991: 2023-03-20 10:58:02,224 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991., storeName=ee10ffd90d917f64b63db6b593dd0991/A, priority=11, startTime=1679309880859; duration=1sec 2023-03-20 10:58:02,224 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:58:02,225 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 10:58:02,225 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36552 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 10:58:02,225 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): ee10ffd90d917f64b63db6b593dd0991/C is initiating minor compaction (all files) 2023-03-20 10:58:02,226 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ee10ffd90d917f64b63db6b593dd0991/C in TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:02,226 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/3ed679e38c8c47d79b39dd77b5989c68, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/1745645435e14d108c602095c57ca9cf, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/c0f6ef5209844c2c9a8dab31926d3764] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp, totalSize=35.7 K 2023-03-20 10:58:02,226 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 3ed679e38c8c47d79b39dd77b5989c68, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1679309879038 2023-03-20 10:58:02,226 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 1745645435e14d108c602095c57ca9cf, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=214, earliestPutTs=1679309879906 2023-03-20 10:58:02,227 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting c0f6ef5209844c2c9a8dab31926d3764, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=233, earliestPutTs=1679309881088 2023-03-20 10:58:02,240 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): ee10ffd90d917f64b63db6b593dd0991#C#compaction#346 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:58:02,250 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/B/3e5c064714b24214923ae32a78d0d86c as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/3e5c064714b24214923ae32a78d0d86c 2023-03-20 10:58:02,257 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ee10ffd90d917f64b63db6b593dd0991/B of ee10ffd90d917f64b63db6b593dd0991 into 3e5c064714b24214923ae32a78d0d86c(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:58:02,257 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ee10ffd90d917f64b63db6b593dd0991: 2023-03-20 10:58:02,257 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991., storeName=ee10ffd90d917f64b63db6b593dd0991/B, priority=13, startTime=1679309882184; duration=0sec 2023-03-20 10:58:02,257 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:58:02,270 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/C/4a16b1810ced4f6f82b05b775ea6f0c1 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/4a16b1810ced4f6f82b05b775ea6f0c1 2023-03-20 10:58:02,276 DEBUG [Thread-2246] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:02,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:02,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:02,282 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ee10ffd90d917f64b63db6b593dd0991/C of ee10ffd90d917f64b63db6b593dd0991 into 4a16b1810ced4f6f82b05b775ea6f0c1(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:58:02,283 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ee10ffd90d917f64b63db6b593dd0991: 2023-03-20 10:58:02,283 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991., storeName=ee10ffd90d917f64b63db6b593dd0991/C, priority=13, startTime=1679309882184; duration=0sec 2023-03-20 10:58:02,283 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:58:02,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:02,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:02,283 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:02,283 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:02,283 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:02,283 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:02,284 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:02,284 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:02,286 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:02,286 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:02,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:02,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:02,287 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:02,287 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:02,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:02,287 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:02,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:02,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:02,293 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:02,295 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:02,295 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:02,297 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:02,297 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:02,297 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:02,300 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:02,300 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:02,300 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:02,300 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. started... 2023-03-20 10:58:02,301 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ee10ffd90d917f64b63db6b593dd0991 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-20 10:58:02,301 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee10ffd90d917f64b63db6b593dd0991, store=A 2023-03-20 10:58:02,301 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:02,301 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee10ffd90d917f64b63db6b593dd0991, store=B 2023-03-20 10:58:02,301 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:02,301 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee10ffd90d917f64b63db6b593dd0991, store=C 2023-03-20 10:58:02,301 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:02,321 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230320878ddeb6c6d94db8b113d7b401d9d231 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320878ddeb6c6d94db8b113d7b401d9d231 2023-03-20 10:58:02,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on ee10ffd90d917f64b63db6b593dd0991 2023-03-20 10:58:02,397 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. as already flushing 2023-03-20 10:58:02,428 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:02,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43528 deadline: 1679309942421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:02,437 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:02,437 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:02,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43530 deadline: 1679309942426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:02,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43494 deadline: 1679309942426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:02,437 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:02,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43478 deadline: 1679309942427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:02,438 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:02,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43492 deadline: 1679309942428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:02,533 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:02,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43528 deadline: 1679309942529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:02,545 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:02,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43494 deadline: 1679309942538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:02,546 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:02,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43530 deadline: 1679309942538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:02,546 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:02,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43478 deadline: 1679309942541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:02,546 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:02,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43492 deadline: 1679309942541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:02,739 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:02,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43528 deadline: 1679309942735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:02,740 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=251, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/A/f898ad23c6b64a669103478af37b965f 2023-03-20 10:58:02,753 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:02,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43494 deadline: 1679309942746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:02,753 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:02,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43478 deadline: 1679309942752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:02,753 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:02,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43492 deadline: 1679309942752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:02,755 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:02,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43530 deadline: 1679309942754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:02,776 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=251 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/B/dced5faefb7e47ff9f6a410cdeb28b0a 2023-03-20 10:58:03,054 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:03,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43528 deadline: 1679309943046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:03,064 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:03,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43530 deadline: 1679309943058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:03,065 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:03,065 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:03,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43492 deadline: 1679309943059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:03,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43478 deadline: 1679309943059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:03,069 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:03,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43494 deadline: 1679309943062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:03,224 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=251 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/C/9730fad3c5d4470db0df5aa740d2a3a9 2023-03-20 10:58:03,228 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/A/f898ad23c6b64a669103478af37b965f as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/f898ad23c6b64a669103478af37b965f 2023-03-20 10:58:03,235 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/f898ad23c6b64a669103478af37b965f, entries=150, sequenceid=251, filesize=25.3 K 2023-03-20 10:58:03,236 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/B/dced5faefb7e47ff9f6a410cdeb28b0a as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/dced5faefb7e47ff9f6a410cdeb28b0a 2023-03-20 10:58:03,240 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/dced5faefb7e47ff9f6a410cdeb28b0a, entries=150, sequenceid=251, filesize=11.8 K 2023-03-20 10:58:03,241 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/C/9730fad3c5d4470db0df5aa740d2a3a9 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/9730fad3c5d4470db0df5aa740d2a3a9 2023-03-20 10:58:03,244 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/9730fad3c5d4470db0df5aa740d2a3a9, entries=150, sequenceid=251, filesize=11.8 K 2023-03-20 10:58:03,244 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for ee10ffd90d917f64b63db6b593dd0991 in 943ms, sequenceid=251, compaction requested=true 2023-03-20 10:58:03,244 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ee10ffd90d917f64b63db6b593dd0991: 2023-03-20 10:58:03,244 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:03,245 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:03,245 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:03,245 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:03,245 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:03,245 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:03,246 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:03,246 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:03,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:03,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:03,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:03,246 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:03,246 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:03,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:03,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:03,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:03,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:03,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:03,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:03,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b1c8e00[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:03,247 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:03,247 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:03,247 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:03,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:03,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:03,248 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:03,248 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:03,248 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:03,248 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:03,248 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:03,248 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:03,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:03,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:03,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:03,249 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:03,249 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:03,249 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:03,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:03,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:03,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:03,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:03,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:03,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:03,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:03,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:03,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:03,252 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:03,253 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:03,253 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:03,253 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:03,253 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:03,254 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:03,254 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:03,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:03,254 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:03,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:03,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:03,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:03,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:03,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:03,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:03,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:03,255 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:03,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:03,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:03,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:03,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:03,257 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:03,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:03,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:03,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:03,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:03,259 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:03,259 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:03,259 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:03,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:03,259 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:03,259 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:03,259 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:03,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:03,259 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:03,259 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:03,259 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:03,259 DEBUG [Thread-2246] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:03,259 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:03,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:03,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:03,259 DEBUG [Thread-2246] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:03,259 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:03,260 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:03,260 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:03,360 DEBUG [Thread-2246] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:03,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:03,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:03,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:03,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:03,362 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:03,362 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:03,362 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:03,362 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:03,363 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:03,363 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:03,366 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:03,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:03,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:03,367 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:03,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:03,367 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:03,367 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:03,367 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:03,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:03,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:03,367 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:03,368 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:03,368 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:03,368 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:03,368 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:03,368 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:03,368 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:03,368 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:03,368 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:03,368 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. started... 2023-03-20 10:58:03,368 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ee10ffd90d917f64b63db6b593dd0991 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-20 10:58:03,368 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee10ffd90d917f64b63db6b593dd0991, store=A 2023-03-20 10:58:03,368 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:03,368 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee10ffd90d917f64b63db6b593dd0991, store=B 2023-03-20 10:58:03,369 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:03,369 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee10ffd90d917f64b63db6b593dd0991, store=C 2023-03-20 10:58:03,369 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:03,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on ee10ffd90d917f64b63db6b593dd0991 2023-03-20 10:58:03,563 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. as already flushing 2023-03-20 10:58:03,589 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:03,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43492 deadline: 1679309943580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:03,590 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:03,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43530 deadline: 1679309943581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:03,594 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:03,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43478 deadline: 1679309943584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:03,594 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:03,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43494 deadline: 1679309943585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:03,594 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:03,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43528 deadline: 1679309943589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:03,695 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:03,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43492 deadline: 1679309943690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:03,695 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:03,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43530 deadline: 1679309943691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:03,699 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:03,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43494 deadline: 1679309943695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:03,699 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:03,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43478 deadline: 1679309943695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:03,699 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:03,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43528 deadline: 1679309943695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:03,779 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230320a857282d19f34afa85a906e5017482c8 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320a857282d19f34afa85a906e5017482c8 2023-03-20 10:58:03,898 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:03,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43492 deadline: 1679309943896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:03,898 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:03,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43530 deadline: 1679309943897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:03,901 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:03,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43478 deadline: 1679309943900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:03,901 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:03,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43494 deadline: 1679309943900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:03,905 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:03,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43528 deadline: 1679309943901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:04,188 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=268, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/A/3ab8e1a10ec84e5795702b4e2b0e11ba 2023-03-20 10:58:04,203 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:04,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43530 deadline: 1679309944200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:04,204 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=268 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/B/0c9aa65593ab42e5bdde7580a43a6679 2023-03-20 10:58:04,207 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:04,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43492 deadline: 1679309944201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:04,207 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:04,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43478 deadline: 1679309944204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:04,208 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:04,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43494 deadline: 1679309944206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:04,213 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:04,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43528 deadline: 1679309944206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:04,222 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=268 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/C/7cd648fa1dd645078b2e5ccfb65fcdaf 2023-03-20 10:58:04,226 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/A/3ab8e1a10ec84e5795702b4e2b0e11ba as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/3ab8e1a10ec84e5795702b4e2b0e11ba 2023-03-20 10:58:04,230 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/3ab8e1a10ec84e5795702b4e2b0e11ba, entries=150, sequenceid=268, filesize=25.4 K 2023-03-20 10:58:04,231 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/B/0c9aa65593ab42e5bdde7580a43a6679 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/0c9aa65593ab42e5bdde7580a43a6679 2023-03-20 10:58:04,233 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/0c9aa65593ab42e5bdde7580a43a6679, entries=150, sequenceid=268, filesize=11.9 K 2023-03-20 10:58:04,234 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/C/7cd648fa1dd645078b2e5ccfb65fcdaf as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/7cd648fa1dd645078b2e5ccfb65fcdaf 2023-03-20 10:58:04,238 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/7cd648fa1dd645078b2e5ccfb65fcdaf, entries=150, sequenceid=268, filesize=11.9 K 2023-03-20 10:58:04,239 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for ee10ffd90d917f64b63db6b593dd0991 in 871ms, sequenceid=268, compaction requested=true 2023-03-20 10:58:04,239 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ee10ffd90d917f64b63db6b593dd0991: 2023-03-20 10:58:04,239 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:04,239 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:04,239 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:04,239 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:04,239 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:04,239 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:04,240 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:04,240 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:04,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:04,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:04,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:04,240 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:04,240 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:04,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:04,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:04,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:04,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:04,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:04,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:04,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7869d303[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:04,241 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:04,241 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:04,242 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:04,242 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:04,242 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:04,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:04,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:04,242 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:04,242 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:04,242 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:04,243 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:04,243 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:04,243 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:04,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:04,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:04,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:04,243 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:04,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:04,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:04,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:04,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:04,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:04,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:04,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:04,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:04,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:04,245 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:04,245 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:04,245 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:04,245 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:04,245 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:04,245 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:04,246 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:04,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:04,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:04,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:04,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:04,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:04,246 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:04,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:04,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:04,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:04,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:04,246 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:04,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:04,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:04,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:04,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:04,247 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:04,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:04,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:04,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:04,249 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:04,249 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:04,249 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:04,249 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:04,249 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:04,249 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:04,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:04,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:04,249 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:04,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:04,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:04,249 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:04,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:04,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:04,249 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:04,249 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:04,249 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:04,249 DEBUG [Thread-2246] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:04,249 DEBUG [Thread-2246] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:04,349 DEBUG [Thread-2246] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:04,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:04,351 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:04,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:04,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:04,352 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:04,352 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:04,352 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:04,352 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:04,352 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:04,352 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:04,353 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:04,353 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:04,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:04,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:04,353 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:04,353 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:04,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:04,353 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:04,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:04,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:04,354 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:04,354 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:04,354 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:04,354 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:04,354 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:04,354 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:04,358 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:04,358 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:04,358 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:04,358 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. started... 2023-03-20 10:58:04,358 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ee10ffd90d917f64b63db6b593dd0991 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-20 10:58:04,358 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee10ffd90d917f64b63db6b593dd0991, store=A 2023-03-20 10:58:04,358 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:04,358 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee10ffd90d917f64b63db6b593dd0991, store=B 2023-03-20 10:58:04,358 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:04,358 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee10ffd90d917f64b63db6b593dd0991, store=C 2023-03-20 10:58:04,358 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:04,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on ee10ffd90d917f64b63db6b593dd0991 2023-03-20 10:58:04,713 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. as already flushing 2023-03-20 10:58:04,742 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:04,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43528 deadline: 1679309944731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:04,742 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:04,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43494 deadline: 1679309944733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:04,743 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:04,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43530 deadline: 1679309944734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:04,748 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:04,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43492 deadline: 1679309944738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:04,748 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:04,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43478 deadline: 1679309944742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:04,779 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303201493c7970db04d1c9aa8d6d9b655db1f to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303201493c7970db04d1c9aa8d6d9b655db1f 2023-03-20 10:58:04,785 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=283, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/A/fb8b8aca2e474df0ab259b7a3658423e 2023-03-20 10:58:04,800 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=283 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/B/0467d0e9f001484eb2a0527a3de86060 2023-03-20 10:58:04,833 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=283 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/C/a69e6341380344a1b13fe9e69e302387 2023-03-20 10:58:04,837 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/A/fb8b8aca2e474df0ab259b7a3658423e as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/fb8b8aca2e474df0ab259b7a3658423e 2023-03-20 10:58:04,841 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/fb8b8aca2e474df0ab259b7a3658423e, entries=150, sequenceid=283, filesize=25.4 K 2023-03-20 10:58:04,841 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/B/0467d0e9f001484eb2a0527a3de86060 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/0467d0e9f001484eb2a0527a3de86060 2023-03-20 10:58:04,846 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/0467d0e9f001484eb2a0527a3de86060, entries=150, sequenceid=283, filesize=11.9 K 2023-03-20 10:58:04,847 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/C/a69e6341380344a1b13fe9e69e302387 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/a69e6341380344a1b13fe9e69e302387 2023-03-20 10:58:04,852 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:04,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43528 deadline: 1679309944843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:04,852 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:04,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43494 deadline: 1679309944843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:04,852 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:04,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43530 deadline: 1679309944843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:04,855 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/a69e6341380344a1b13fe9e69e302387, entries=150, sequenceid=283, filesize=11.9 K 2023-03-20 10:58:04,856 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for ee10ffd90d917f64b63db6b593dd0991 in 498ms, sequenceid=283, compaction requested=true 2023-03-20 10:58:04,856 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ee10ffd90d917f64b63db6b593dd0991: 2023-03-20 10:58:04,856 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:04,856 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:04,856 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:04,856 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:04,856 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:04,856 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:04,857 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:04,857 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:04,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:04,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:04,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:04,858 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:04,858 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:04,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:04,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:04,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:04,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:04,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on ee10ffd90d917f64b63db6b593dd0991 2023-03-20 10:58:04,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:04,859 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ee10ffd90d917f64b63db6b593dd0991 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-20 10:58:04,860 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:04,860 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f0204cf[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:04,860 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:04,860 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:04,860 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee10ffd90d917f64b63db6b593dd0991, store=A 2023-03-20 10:58:04,860 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:04,860 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee10ffd90d917f64b63db6b593dd0991, store=B 2023-03-20 10:58:04,860 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:04,860 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee10ffd90d917f64b63db6b593dd0991, store=C 2023-03-20 10:58:04,860 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:04,860 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:04,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:04,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:04,861 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:04,861 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:04,861 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:04,861 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:04,861 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:04,861 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:04,861 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:04,861 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:04,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:04,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:04,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:04,861 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:04,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:04,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:04,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:04,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:04,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:04,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:04,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:04,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:04,863 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:04,863 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:04,864 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:04,864 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:04,864 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:04,864 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:04,864 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:04,864 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:04,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:04,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:04,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:04,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:04,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:04,865 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:04,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:04,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:04,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:04,865 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:04,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:04,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:04,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:04,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:04,866 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:04,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:04,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:04,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:04,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:04,878 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:04,878 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:04,878 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:04,878 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:04,878 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:04,878 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:04,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:04,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:04,878 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:04,878 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:04,878 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:04,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:04,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:04,879 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:04,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:04,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:04,879 DEBUG [Thread-2246] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:04,879 DEBUG [Thread-2246] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:04,916 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:04,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43492 deadline: 1679309944905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:04,917 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:04,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43478 deadline: 1679309944910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:04,979 DEBUG [Thread-2246] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:04,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:04,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:04,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:04,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:04,985 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:04,985 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:04,985 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:04,985 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:04,986 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:04,986 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:04,986 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:04,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:04,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:04,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:04,987 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:04,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:04,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:04,987 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:04,988 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:04,988 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:04,988 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:04,988 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:04,988 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:04,988 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:04,988 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:04,988 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:04,988 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. started... 2023-03-20 10:58:04,988 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. as already flushing 2023-03-20 10:58:04,988 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:04,988 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:04,989 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:04,989 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:04,989 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:04,989 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:04,989 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:04,990 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:04,990 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:04,990 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:04,990 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:04,990 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:04,990 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46bce651[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:04,990 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:04,990 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:04,990 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:04,991 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:04,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:04,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:04,991 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:04,991 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:04,991 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:04,991 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:04,991 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:04,992 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:04,992 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:04,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:04,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:04,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:04,992 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:04,992 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:04,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:04,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:04,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:04,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:04,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:04,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:04,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:04,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:04,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:04,994 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:04,994 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:04,994 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:04,994 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:04,994 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:04,996 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:04,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:04,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:04,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:04,996 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:04,996 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:04,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:04,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:04,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:04,996 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:04,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:04,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:04,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:04,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:04,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:04,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:04,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:04,997 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:04,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:04,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:04,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:04,999 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:04,999 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:04,999 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:04,999 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:04,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:04,999 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:04,999 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:04,999 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:04,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:04,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:04,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:04,999 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:04,999 DEBUG [Thread-2246] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:04,999 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:04,999 DEBUG [Thread-2246] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:04,999 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:04,999 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:04,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:04,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:05,020 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:05,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43478 deadline: 1679309945018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,027 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:05,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43492 deadline: 1679309945019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,056 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:05,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43528 deadline: 1679309945054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,068 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:05,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43530 deadline: 1679309945056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,068 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:05,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43494 deadline: 1679309945056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,099 DEBUG [Thread-2246] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:05,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:05,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:05,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:05,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:05,106 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:05,106 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:05,106 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:05,106 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:05,106 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:05,106 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:05,107 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:05,107 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:05,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:05,107 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,107 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:05,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:05,108 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:05,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:05,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:05,108 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:05,114 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:05,114 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:05,114 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:05,114 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:05,114 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:05,114 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:05,114 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:05,114 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:05,114 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. started... 2023-03-20 10:58:05,114 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. as already flushing 2023-03-20 10:58:05,115 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:05,115 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:05,115 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:05,115 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,115 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:05,115 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:05,116 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,116 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:05,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:05,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:05,117 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:05,117 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:05,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:05,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:05,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:05,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:05,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:05,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f86f9ee[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:05,118 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:05,118 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:05,127 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:05,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:05,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:05,128 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:05,128 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:05,128 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:05,129 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:05,129 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,129 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:05,129 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:05,129 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:05,129 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:05,129 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:05,129 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:05,129 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:05,129 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:05,129 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:05,130 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:05,130 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:05,130 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:05,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:05,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:05,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:05,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:05,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:05,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:05,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:05,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:05,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:05,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:05,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:05,132 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:05,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:05,133 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:05,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,134 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,134 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:05,134 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:05,134 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,134 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:05,134 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:05,134 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:05,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:05,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:05,134 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:05,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:05,134 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:05,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:05,134 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:05,134 DEBUG [Thread-2246] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:05,134 DEBUG [Thread-2246] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:05,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:05,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:05,227 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:05,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43478 deadline: 1679309945223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,228 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:05,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43492 deadline: 1679309945228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,235 DEBUG [Thread-2246] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:05,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:05,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:05,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:05,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:05,239 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:05,240 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:05,240 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:05,240 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:05,240 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:05,240 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:05,241 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:05,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:05,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:05,241 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,241 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,241 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:05,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:05,242 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:05,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:05,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:05,242 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:05,243 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:05,243 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:05,243 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:05,243 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:05,243 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:05,250 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:05,250 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:05,250 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:05,250 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. started... 2023-03-20 10:58:05,250 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. as already flushing 2023-03-20 10:58:05,250 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:05,250 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:05,250 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:05,250 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,250 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:05,250 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:05,252 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,252 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:05,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:05,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:05,252 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:05,252 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:05,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:05,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:05,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:05,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:05,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:05,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30451890[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:05,253 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:05,253 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:05,254 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:05,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:05,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:05,254 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:05,254 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:05,254 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,254 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:05,254 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:05,255 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:05,255 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,255 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:05,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:05,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:05,255 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:05,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:05,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:05,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:05,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:05,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:05,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:05,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:05,258 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:05,258 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:05,258 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:05,258 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:05,258 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:05,258 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:05,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:05,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:05,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:05,259 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:05,259 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:05,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:05,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:05,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:05,261 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:05,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:05,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:05,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:05,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:05,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:05,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:05,264 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:05,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,265 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:05,265 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:05,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:05,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:05,265 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,265 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:05,265 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:05,265 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,265 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:05,266 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:05,266 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:05,266 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:05,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:05,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:05,266 DEBUG [Thread-2246] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:05,266 DEBUG [Thread-2246] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:05,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:05,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:05,299 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230320fca12e0f419c41019753b9aec808f72b to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320fca12e0f419c41019753b9aec808f72b 2023-03-20 10:58:05,310 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=302, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/A/d6de0dc7ad8c431e9ddc0d4a942fd140 2023-03-20 10:58:05,364 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:05,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43528 deadline: 1679309945358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,366 DEBUG [Thread-2246] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:05,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:05,378 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:05,378 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:05,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43530 deadline: 1679309945373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,378 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:05,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43494 deadline: 1679309945375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:05,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:05,382 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:05,382 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:05,382 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:05,382 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:05,383 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:05,383 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:05,383 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:05,383 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:05,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:05,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:05,384 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:05,384 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,384 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:05,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:05,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:05,385 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:05,395 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:05,395 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:05,395 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:05,395 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:05,395 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:05,396 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:05,396 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:05,396 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:05,396 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. started... 2023-03-20 10:58:05,396 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. as already flushing 2023-03-20 10:58:05,396 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:05,396 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:05,396 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:05,396 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,396 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:05,396 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:05,397 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,397 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:05,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:05,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:05,397 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:05,397 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:05,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:05,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:05,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:05,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:05,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:05,399 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:05,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6023450[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:05,399 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:05,400 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:05,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:05,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:05,401 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:05,401 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:05,401 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:05,401 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,401 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:05,401 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,402 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:05,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:05,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:05,402 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:05,402 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:05,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:05,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:05,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:05,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:05,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:05,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:05,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:05,404 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:05,404 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:05,404 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:05,404 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:05,404 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:05,406 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:05,406 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:05,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:05,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:05,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:05,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:05,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:05,406 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:05,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:05,406 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:05,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:05,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:05,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:05,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:05,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:05,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,408 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:05,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:05,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,409 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:05,409 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:05,409 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:05,409 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:05,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:05,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:05,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:05,409 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:05,410 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:05,410 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,410 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:05,410 DEBUG [Thread-2246] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:05,410 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:05,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:05,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:05,410 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:05,410 DEBUG [Thread-2246] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:05,514 DEBUG [Thread-2246] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:05,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:05,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:05,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:05,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:05,517 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:05,517 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:05,517 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:05,517 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:05,517 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:05,517 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:05,525 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,525 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:05,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:05,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:05,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:05,526 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,526 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:05,526 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:05,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:05,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:05,527 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:05,534 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:05,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43478 deadline: 1679309945529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,541 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:05,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43492 deadline: 1679309945533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,550 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:05,550 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:05,550 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:05,550 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:05,550 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:05,550 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:05,550 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:05,550 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:05,550 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. started... 2023-03-20 10:58:05,550 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. as already flushing 2023-03-20 10:58:05,551 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:05,551 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:05,551 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:05,551 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,551 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:05,551 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:05,552 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:05,552 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:05,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:05,552 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:05,553 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:05,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:05,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:05,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:05,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:05,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:05,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29096c23[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:05,555 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:05,555 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:05,556 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:05,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:05,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:05,556 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:05,556 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:05,556 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:05,556 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,556 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:05,557 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,557 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:05,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:05,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:05,557 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:05,557 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:05,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:05,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:05,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:05,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:05,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:05,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:05,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:05,560 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:05,560 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:05,560 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:05,560 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:05,561 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:05,561 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:05,561 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:05,561 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:05,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:05,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:05,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:05,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:05,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:05,562 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:05,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:05,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:05,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:05,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:05,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:05,563 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:05,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:05,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:05,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,566 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:05,566 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:05,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:05,566 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:05,566 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:05,566 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:05,566 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,566 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:05,566 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:05,566 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:05,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:05,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:05,568 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:05,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:05,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:05,568 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:05,570 DEBUG [Thread-2246] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:05,570 DEBUG [Thread-2246] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:05,670 DEBUG [Thread-2246] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:05,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:05,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:05,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:05,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:05,673 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:05,673 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:05,673 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:05,673 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:05,673 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:05,674 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:05,675 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,675 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:05,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:05,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:05,675 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,675 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:05,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:05,675 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:05,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:05,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:05,675 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:05,676 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:05,676 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:05,676 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:05,676 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:05,676 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:05,676 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:05,676 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:05,676 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:05,676 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. started... 2023-03-20 10:58:05,676 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. as already flushing 2023-03-20 10:58:05,676 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:05,676 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:05,676 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:05,676 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,676 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:05,676 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:05,677 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,677 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:05,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:05,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:05,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:05,678 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:05,678 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:05,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:05,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:05,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:05,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:05,679 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:05,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@168a965[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:05,679 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:05,680 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:05,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:05,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:05,680 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:05,680 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:05,680 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:05,680 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,680 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:05,681 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:05,681 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:05,681 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,681 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:05,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:05,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:05,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:05,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:05,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:05,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:05,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:05,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:05,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:05,683 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:05,684 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:05,684 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:05,684 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:05,684 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:05,685 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:05,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:05,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:05,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:05,685 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:05,685 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:05,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:05,685 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:05,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:05,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:05,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:05,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:05,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:05,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:05,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:05,686 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:05,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:05,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,695 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:05,695 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:05,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:05,695 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:05,695 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:05,695 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:05,695 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:05,695 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,695 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:05,695 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:05,695 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:05,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:05,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:05,696 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:05,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:05,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:05,696 DEBUG [Thread-2246] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:05,696 DEBUG [Thread-2246] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:05,734 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=302 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/B/dfaf99f8655542d28dbaec0be6d52bac 2023-03-20 10:58:05,759 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=302 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/C/07c6d6c76aba4853b007d6e3868a0ed4 2023-03-20 10:58:05,763 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/A/d6de0dc7ad8c431e9ddc0d4a942fd140 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/d6de0dc7ad8c431e9ddc0d4a942fd140 2023-03-20 10:58:05,769 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/d6de0dc7ad8c431e9ddc0d4a942fd140, entries=250, sequenceid=302, filesize=39.2 K 2023-03-20 10:58:05,770 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/B/dfaf99f8655542d28dbaec0be6d52bac as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/dfaf99f8655542d28dbaec0be6d52bac 2023-03-20 10:58:05,774 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/dfaf99f8655542d28dbaec0be6d52bac, entries=150, sequenceid=302, filesize=11.9 K 2023-03-20 10:58:05,775 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/C/07c6d6c76aba4853b007d6e3868a0ed4 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/07c6d6c76aba4853b007d6e3868a0ed4 2023-03-20 10:58:05,791 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/07c6d6c76aba4853b007d6e3868a0ed4, entries=150, sequenceid=302, filesize=11.9 K 2023-03-20 10:58:05,792 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for ee10ffd90d917f64b63db6b593dd0991 in 933ms, sequenceid=302, compaction requested=true 2023-03-20 10:58:05,792 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ee10ffd90d917f64b63db6b593dd0991: 2023-03-20 10:58:05,792 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:58:05,792 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-03-20 10:58:05,792 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-20 10:58:05,792 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:58:05,792 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:58:05,794 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61176 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-20 10:58:05,794 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): ee10ffd90d917f64b63db6b593dd0991/B is initiating minor compaction (all files) 2023-03-20 10:58:05,794 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ee10ffd90d917f64b63db6b593dd0991/B in TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:05,794 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/3e5c064714b24214923ae32a78d0d86c, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/dced5faefb7e47ff9f6a410cdeb28b0a, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/0c9aa65593ab42e5bdde7580a43a6679, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/0467d0e9f001484eb2a0527a3de86060, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/dfaf99f8655542d28dbaec0be6d52bac] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp, totalSize=59.7 K 2023-03-20 10:58:05,795 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 3e5c064714b24214923ae32a78d0d86c, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=233, earliestPutTs=1679309881088 2023-03-20 10:58:05,795 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 203240 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-03-20 10:58:05,795 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1912): ee10ffd90d917f64b63db6b593dd0991/A is initiating minor compaction (all files) 2023-03-20 10:58:05,795 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ee10ffd90d917f64b63db6b593dd0991/A in TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:05,795 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/45da4837dcb74f89abc1e30f6855f2a0, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/26340732b60145d1af43001b00da6166, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/98228da5c6584de4b6fffc4e081c9fbf, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/f898ad23c6b64a669103478af37b965f, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/3ab8e1a10ec84e5795702b4e2b0e11ba, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/fb8b8aca2e474df0ab259b7a3658423e, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/d6de0dc7ad8c431e9ddc0d4a942fd140] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp, totalSize=198.5 K 2023-03-20 10:58:05,795 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting dced5faefb7e47ff9f6a410cdeb28b0a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1679309881240 2023-03-20 10:58:05,796 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 0c9aa65593ab42e5bdde7580a43a6679, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1679309882413 2023-03-20 10:58:05,796 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 45da4837dcb74f89abc1e30f6855f2a0, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1679309879038 2023-03-20 10:58:05,796 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 0467d0e9f001484eb2a0527a3de86060, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=283, earliestPutTs=1679309883580 2023-03-20 10:58:05,796 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 26340732b60145d1af43001b00da6166, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=214, earliestPutTs=1679309879906 2023-03-20 10:58:05,796 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 98228da5c6584de4b6fffc4e081c9fbf, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=233, earliestPutTs=1679309881077 2023-03-20 10:58:05,796 DEBUG [Thread-2246] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:05,797 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting dfaf99f8655542d28dbaec0be6d52bac, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=302, earliestPutTs=1679309884728 2023-03-20 10:58:05,797 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting f898ad23c6b64a669103478af37b965f, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1679309881240 2023-03-20 10:58:05,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:05,798 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 3ab8e1a10ec84e5795702b4e2b0e11ba, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1679309882413 2023-03-20 10:58:05,799 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting fb8b8aca2e474df0ab259b7a3658423e, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=283, earliestPutTs=1679309883580 2023-03-20 10:58:05,799 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting d6de0dc7ad8c431e9ddc0d4a942fd140, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=302, earliestPutTs=1679309884728 2023-03-20 10:58:05,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:05,816 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): ee10ffd90d917f64b63db6b593dd0991#B#compaction#359 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:58:05,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:05,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:05,817 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:05,817 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:05,817 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:05,817 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:05,817 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:05,817 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:05,818 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:05,818 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:05,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:05,818 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:05,818 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:05,819 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:05,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:05,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:05,819 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:05,830 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:05,830 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:05,830 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:05,830 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:05,830 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:05,834 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:05,834 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:05,834 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:05,834 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. started... 2023-03-20 10:58:05,834 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ee10ffd90d917f64b63db6b593dd0991 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-20 10:58:05,834 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee10ffd90d917f64b63db6b593dd0991, store=A 2023-03-20 10:58:05,836 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:05,836 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee10ffd90d917f64b63db6b593dd0991, store=B 2023-03-20 10:58:05,836 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:05,836 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee10ffd90d917f64b63db6b593dd0991, store=C 2023-03-20 10:58:05,836 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:05,842 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] throttle.PressureAwareThroughputController(145): ee10ffd90d917f64b63db6b593dd0991#A#compaction#360 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:58:05,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on ee10ffd90d917f64b63db6b593dd0991 2023-03-20 10:58:05,892 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. as already flushing 2023-03-20 10:58:05,982 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:05,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43494 deadline: 1679309945974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,983 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:05,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43530 deadline: 1679309945974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:05,983 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:05,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43528 deadline: 1679309945976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:06,049 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:06,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43478 deadline: 1679309946041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:06,049 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:06,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43492 deadline: 1679309946043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:06,087 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:06,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43494 deadline: 1679309946083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:06,087 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:06,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43530 deadline: 1679309946083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:06,088 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:06,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43528 deadline: 1679309946084, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:06,251 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/B/1497d0653d3b4923b056212d696b8940 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/1497d0653d3b4923b056212d696b8940 2023-03-20 10:58:06,257 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in ee10ffd90d917f64b63db6b593dd0991/B of ee10ffd90d917f64b63db6b593dd0991 into 1497d0653d3b4923b056212d696b8940(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:58:06,257 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ee10ffd90d917f64b63db6b593dd0991: 2023-03-20 10:58:06,257 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991., storeName=ee10ffd90d917f64b63db6b593dd0991/B, priority=11, startTime=1679309885792; duration=0sec 2023-03-20 10:58:06,257 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:58:06,257 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-20 10:58:06,258 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61176 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-20 10:58:06,259 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): ee10ffd90d917f64b63db6b593dd0991/C is initiating minor compaction (all files) 2023-03-20 10:58:06,259 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ee10ffd90d917f64b63db6b593dd0991/C in TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:06,259 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/4a16b1810ced4f6f82b05b775ea6f0c1, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/9730fad3c5d4470db0df5aa740d2a3a9, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/7cd648fa1dd645078b2e5ccfb65fcdaf, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/a69e6341380344a1b13fe9e69e302387, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/07c6d6c76aba4853b007d6e3868a0ed4] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp, totalSize=59.7 K 2023-03-20 10:58:06,259 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 4a16b1810ced4f6f82b05b775ea6f0c1, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=233, earliestPutTs=1679309881088 2023-03-20 10:58:06,260 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 9730fad3c5d4470db0df5aa740d2a3a9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1679309881240 2023-03-20 10:58:06,260 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 7cd648fa1dd645078b2e5ccfb65fcdaf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=268, earliestPutTs=1679309882413 2023-03-20 10:58:06,260 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting a69e6341380344a1b13fe9e69e302387, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=283, earliestPutTs=1679309883580 2023-03-20 10:58:06,260 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 07c6d6c76aba4853b007d6e3868a0ed4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=302, earliestPutTs=1679309884728 2023-03-20 10:58:06,269 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): ee10ffd90d917f64b63db6b593dd0991#C#compaction#362 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:58:06,291 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303208e0ec7b9735949e4b91338fc7f623d18 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303208e0ec7b9735949e4b91338fc7f623d18 2023-03-20 10:58:06,294 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:06,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43528 deadline: 1679309946289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:06,296 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/C/61400ef950584afbad889a7bba056380 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/61400ef950584afbad889a7bba056380 2023-03-20 10:58:06,298 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:06,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43494 deadline: 1679309946289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:06,299 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:06,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43530 deadline: 1679309946289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:06,300 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in ee10ffd90d917f64b63db6b593dd0991/C of ee10ffd90d917f64b63db6b593dd0991 into 61400ef950584afbad889a7bba056380(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:58:06,301 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ee10ffd90d917f64b63db6b593dd0991: 2023-03-20 10:58:06,301 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991., storeName=ee10ffd90d917f64b63db6b593dd0991/C, priority=11, startTime=1679309885792; duration=0sec 2023-03-20 10:58:06,301 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/A/8a7825e93ad940298e0e342d22ebba20 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/8a7825e93ad940298e0e342d22ebba20 2023-03-20 10:58:06,301 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:58:06,305 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=315, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/A/a14207d61cfd46e5ab8cef7cfc72aa59 2023-03-20 10:58:06,307 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in ee10ffd90d917f64b63db6b593dd0991/A of ee10ffd90d917f64b63db6b593dd0991 into 8a7825e93ad940298e0e342d22ebba20(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:58:06,307 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ee10ffd90d917f64b63db6b593dd0991: 2023-03-20 10:58:06,307 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991., storeName=ee10ffd90d917f64b63db6b593dd0991/A, priority=9, startTime=1679309885792; duration=0sec 2023-03-20 10:58:06,307 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:58:06,603 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:06,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43528 deadline: 1679309946597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:06,606 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:06,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43494 deadline: 1679309946601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:06,606 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:06,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43530 deadline: 1679309946601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:06,743 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=315 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/B/b54a5a0d03c34499ae1e1b7d2898b3c6 2023-03-20 10:58:07,053 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:07,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43478 deadline: 1679309947050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:07,054 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:07,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43492 deadline: 1679309947052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:07,106 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:07,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43528 deadline: 1679309947104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:07,110 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:07,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43530 deadline: 1679309947108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:07,111 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:07,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43494 deadline: 1679309947108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:07,175 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=315 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/C/e47eae7095be44658d743e29ca81b49e 2023-03-20 10:58:07,180 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/A/a14207d61cfd46e5ab8cef7cfc72aa59 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/a14207d61cfd46e5ab8cef7cfc72aa59 2023-03-20 10:58:07,184 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/a14207d61cfd46e5ab8cef7cfc72aa59, entries=100, sequenceid=315, filesize=18.6 K 2023-03-20 10:58:07,187 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/B/b54a5a0d03c34499ae1e1b7d2898b3c6 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/b54a5a0d03c34499ae1e1b7d2898b3c6 2023-03-20 10:58:07,191 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/b54a5a0d03c34499ae1e1b7d2898b3c6, entries=100, sequenceid=315, filesize=9.5 K 2023-03-20 10:58:07,192 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/C/e47eae7095be44658d743e29ca81b49e as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/e47eae7095be44658d743e29ca81b49e 2023-03-20 10:58:07,228 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/e47eae7095be44658d743e29ca81b49e, entries=100, sequenceid=315, filesize=9.5 K 2023-03-20 10:58:07,229 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for ee10ffd90d917f64b63db6b593dd0991 in 1395ms, sequenceid=315, compaction requested=false 2023-03-20 10:58:07,229 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ee10ffd90d917f64b63db6b593dd0991: 2023-03-20 10:58:07,229 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:07,229 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:07,229 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:07,229 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:07,229 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:07,229 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:07,231 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:07,231 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:07,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:07,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:07,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:07,231 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:07,231 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:07,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:07,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:07,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:07,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:07,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:07,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:07,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1971b040[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:07,243 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:07,243 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:07,245 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:07,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:07,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:07,245 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:07,245 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:07,245 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:07,247 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:07,247 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:07,247 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:07,247 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:07,247 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:07,247 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:07,247 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:07,247 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:07,247 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:07,247 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:07,247 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:07,249 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:07,249 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:07,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:07,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:07,249 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:07,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:07,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:07,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:07,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:07,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:07,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:07,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:07,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:07,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:07,253 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:07,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:07,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:07,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:07,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:07,254 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:07,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:07,262 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:07,262 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:07,262 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:07,262 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:07,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:07,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:07,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:07,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:07,262 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:07,262 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:07,262 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:07,262 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:07,262 DEBUG [Thread-2246] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:07,262 DEBUG [Thread-2246] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:07,262 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:07,262 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:07,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:07,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:07,363 DEBUG [Thread-2246] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:07,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:07,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:07,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:07,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:07,387 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:07,387 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:07,387 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:07,387 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:07,387 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:07,387 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:07,388 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:07,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:07,388 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:07,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:07,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:07,388 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:07,389 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:07,389 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:07,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:07,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:07,389 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:07,389 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:07,389 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:07,389 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:07,389 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:07,390 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:07,390 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:07,390 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:07,390 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:07,390 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. started... 2023-03-20 10:58:07,390 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ee10ffd90d917f64b63db6b593dd0991 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-20 10:58:07,390 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee10ffd90d917f64b63db6b593dd0991, store=A 2023-03-20 10:58:07,390 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:07,390 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee10ffd90d917f64b63db6b593dd0991, store=B 2023-03-20 10:58:07,391 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:07,391 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee10ffd90d917f64b63db6b593dd0991, store=C 2023-03-20 10:58:07,391 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:07,419 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032009570ca0671d421e88b6416d5e3d71dd to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032009570ca0671d421e88b6416d5e3d71dd 2023-03-20 10:58:07,445 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=337, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/A/644f36b2ec244115a01a09d35f4d494d 2023-03-20 10:58:07,876 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=337 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/B/a489a9fe66bf443794dce0a1e72eaaea 2023-03-20 10:58:08,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on ee10ffd90d917f64b63db6b593dd0991 2023-03-20 10:58:08,113 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. as already flushing 2023-03-20 10:58:08,131 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:08,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43494 deadline: 1679309948126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,140 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:08,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43528 deadline: 1679309948131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,140 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:08,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43530 deadline: 1679309948131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,236 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:08,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43494 deadline: 1679309948232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,245 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:08,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43528 deadline: 1679309948241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,245 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:08,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:43530 deadline: 1679309948241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,284 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=337 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/C/f1d93ff160294bec977c9b7ba8a2c688 2023-03-20 10:58:08,289 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/A/644f36b2ec244115a01a09d35f4d494d as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/644f36b2ec244115a01a09d35f4d494d 2023-03-20 10:58:08,300 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/644f36b2ec244115a01a09d35f4d494d, entries=150, sequenceid=337, filesize=25.4 K 2023-03-20 10:58:08,301 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/B/a489a9fe66bf443794dce0a1e72eaaea as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/a489a9fe66bf443794dce0a1e72eaaea 2023-03-20 10:58:08,304 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/a489a9fe66bf443794dce0a1e72eaaea, entries=150, sequenceid=337, filesize=11.9 K 2023-03-20 10:58:08,305 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/C/f1d93ff160294bec977c9b7ba8a2c688 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/f1d93ff160294bec977c9b7ba8a2c688 2023-03-20 10:58:08,314 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/f1d93ff160294bec977c9b7ba8a2c688, entries=150, sequenceid=337, filesize=11.9 K 2023-03-20 10:58:08,315 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for ee10ffd90d917f64b63db6b593dd0991 in 925ms, sequenceid=337, compaction requested=true 2023-03-20 10:58:08,315 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ee10ffd90d917f64b63db6b593dd0991: 2023-03-20 10:58:08,315 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:08,315 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:08,315 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:08,315 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,315 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:08,315 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:08,316 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:08,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:08,316 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:08,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:08,317 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:08,317 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:08,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:08,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:08,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:08,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:08,318 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:08,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ec6b78a[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:08,318 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:08,319 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:08,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:08,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:08,319 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:08,319 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:08,319 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:08,321 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:08,321 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:08,321 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:08,322 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,322 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:08,322 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:08,322 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:08,322 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:08,322 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:08,322 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:08,322 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:08,323 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:08,323 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:08,323 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:08,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:08,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:08,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:08,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:08,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:08,323 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:08,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:08,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:08,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:08,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:08,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:08,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:08,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:08,331 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:08,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,336 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:08,336 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:08,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:08,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:08,336 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:08,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:08,336 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:08,336 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:08,337 DEBUG [Thread-2246] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:08,337 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:08,337 DEBUG [Thread-2246] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:08,337 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,337 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:08,337 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:08,337 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:08,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:08,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:08,437 DEBUG [Thread-2246] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:08,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:08,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:08,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on ee10ffd90d917f64b63db6b593dd0991 2023-03-20 10:58:08,443 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ee10ffd90d917f64b63db6b593dd0991 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-20 10:58:08,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:08,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:08,443 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee10ffd90d917f64b63db6b593dd0991, store=A 2023-03-20 10:58:08,443 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:08,443 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:08,443 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:08,443 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee10ffd90d917f64b63db6b593dd0991, store=B 2023-03-20 10:58:08,443 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:08,443 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:08,443 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:08,443 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee10ffd90d917f64b63db6b593dd0991, store=C 2023-03-20 10:58:08,443 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:08,444 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:08,444 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:08,444 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:08,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:08,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:08,444 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:08,445 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,445 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:08,445 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:08,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:08,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:08,445 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:08,446 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:08,446 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:08,446 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:08,446 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:08,446 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:08,446 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:08,446 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:08,446 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:08,446 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. started... 2023-03-20 10:58:08,446 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. as already flushing 2023-03-20 10:58:08,447 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:08,447 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:08,447 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:08,447 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,447 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:08,447 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:08,448 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:08,449 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:08,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:08,450 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:08,450 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:08,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:08,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:08,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:08,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:08,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:08,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6cccba05[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:08,451 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:08,451 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:08,452 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:08,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:08,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:08,452 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:08,452 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:08,452 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:08,452 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,452 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:08,453 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:08,453 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:08,453 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:08,453 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:08,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:08,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:08,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:08,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:08,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:08,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:08,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:08,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:08,456 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:08,456 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:08,456 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:08,456 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:08,456 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:08,457 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:08,457 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:08,457 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:08,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:08,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:08,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:08,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:08,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:08,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:08,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:08,457 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:08,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:08,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:08,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:08,459 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:08,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:08,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:08,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,461 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,461 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:08,461 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:08,461 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:08,461 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,461 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:08,461 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:08,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:08,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:08,461 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:08,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:08,461 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:08,461 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:08,461 DEBUG [Thread-2246] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:08,461 DEBUG [Thread-2246] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:08,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:08,461 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:08,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:08,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:08,470 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230320793e8ec8c7c643d0818ee0c079c6d612 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320793e8ec8c7c643d0818ee0c079c6d612 2023-03-20 10:58:08,480 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:08,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43494 deadline: 1679309948476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,484 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:08,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43530 deadline: 1679309948476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,484 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:08,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43528 deadline: 1679309948477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,561 DEBUG [Thread-2246] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:08,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:08,563 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:08,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:08,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:08,567 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:08,567 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:08,567 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:08,567 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:08,568 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:08,568 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:08,568 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:08,568 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:08,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:08,569 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,569 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:08,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:08,569 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:08,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:08,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:08,569 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:08,569 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:08,570 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:08,570 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:08,570 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:08,570 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:08,570 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:08,570 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:08,570 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:08,570 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. started... 2023-03-20 10:58:08,570 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. as already flushing 2023-03-20 10:58:08,570 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:08,570 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:08,570 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:08,570 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,570 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:08,570 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:08,571 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,571 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:08,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:08,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:08,571 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:08,571 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:08,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:08,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:08,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:08,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:08,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:08,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28ab0578[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:08,572 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:08,572 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:08,573 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:08,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:08,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:08,573 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:08,573 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,573 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:08,573 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:08,573 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:08,573 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,574 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:08,574 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:08,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:08,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:08,574 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:08,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:08,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:08,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:08,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:08,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:08,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:08,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:08,578 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:08,578 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:08,578 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:08,578 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:08,578 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:08,579 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:08,579 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:08,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:08,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:08,580 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:08,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:08,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:08,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:08,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:08,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:08,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:08,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:08,580 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:08,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:08,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:08,582 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:08,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:08,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,585 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:08,585 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:08,585 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,585 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:08,585 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:08,585 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,585 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:08,585 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:08,585 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:08,585 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:08,585 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:08,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:08,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:08,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:08,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:08,586 DEBUG [Thread-2246] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:08,586 DEBUG [Thread-2246] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:08,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:08,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:08,587 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:08,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43494 deadline: 1679309948582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,587 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:08,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43530 deadline: 1679309948585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,590 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:08,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43528 deadline: 1679309948586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,686 DEBUG [Thread-2246] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:08,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:08,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:08,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:08,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:08,694 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:08,694 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:08,694 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:08,694 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:08,695 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:08,695 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:08,698 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:08,698 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:08,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:08,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:08,698 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,698 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:08,698 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:08,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:08,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:08,699 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:08,699 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:08,699 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:08,699 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:08,699 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:08,699 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:08,699 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:08,699 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:08,699 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:08,700 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. started... 2023-03-20 10:58:08,700 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. as already flushing 2023-03-20 10:58:08,700 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:08,700 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:08,700 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:08,700 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,700 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:08,700 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:08,709 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:08,709 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:08,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:08,709 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:08,709 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:08,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:08,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:08,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:08,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:08,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:08,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c4c539d[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:08,717 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:08,717 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:08,720 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,720 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:08,720 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:08,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:08,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:08,720 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:08,720 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:08,720 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:08,721 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:08,721 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:08,721 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:08,721 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:08,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:08,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:08,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:08,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:08,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:08,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:08,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:08,725 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:08,725 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:08,725 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:08,725 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:08,725 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:08,725 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:08,734 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:08,734 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:08,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:08,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:08,734 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:08,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:08,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:08,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:08,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:08,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:08,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:08,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:08,735 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:08,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:08,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:08,738 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:08,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:08,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,744 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:08,744 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:08,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:08,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:08,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:08,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:08,745 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,745 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:08,745 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:08,745 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,745 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:08,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:08,745 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:08,745 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:08,746 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:08,746 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:08,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:08,746 DEBUG [Thread-2246] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:08,746 DEBUG [Thread-2246] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:08,792 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:08,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43530 deadline: 1679309948788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,792 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:08,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43494 deadline: 1679309948788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,796 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:08,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43528 deadline: 1679309948792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,846 DEBUG [Thread-2246] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:08,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:08,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:08,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:08,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:08,862 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:08,862 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:08,862 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:08,862 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:08,863 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:08,863 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:08,864 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:08,864 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:08,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:08,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:08,865 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:08,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:08,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:08,865 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:08,865 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,866 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:08,875 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:08,875 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:08,875 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:08,875 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:08,875 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:08,875 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:08,875 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:08,875 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:08,875 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. started... 2023-03-20 10:58:08,875 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. as already flushing 2023-03-20 10:58:08,875 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:08,875 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:08,875 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:08,875 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,876 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:08,876 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:08,878 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,878 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:08,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:08,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:08,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:08,878 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:08,878 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:08,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:08,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:08,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:08,879 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=352, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/A/8b6aabe042a54623b1cd44fb5440e753 2023-03-20 10:58:08,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:08,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78dd979[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:08,880 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:08,880 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:08,881 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:08,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:08,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:08,881 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:08,881 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:08,881 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:08,881 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,881 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:08,882 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:08,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:08,883 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:08,883 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:08,883 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:08,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:08,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:08,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:08,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:08,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:08,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:08,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:08,885 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:08,885 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:08,885 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:08,885 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:08,885 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:08,886 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:08,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:08,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:08,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:08,886 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:08,886 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:08,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:08,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:08,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:08,886 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:08,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:08,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:08,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:08,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:08,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:08,887 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:08,889 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=352 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/B/225de2be28a3471381649b5df8ef51ac 2023-03-20 10:58:08,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:08,895 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,895 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:08,895 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:08,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:08,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:08,895 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:08,895 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:08,895 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:08,895 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:08,895 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:08,895 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:08,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:08,895 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:08,895 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:08,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:08,895 DEBUG [Thread-2246] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:08,895 DEBUG [Thread-2246] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:08,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:08,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:08,909 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=352 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/C/4c2ccb73beeb4b0780bd2561ab2c58b8 2023-03-20 10:58:08,913 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/A/8b6aabe042a54623b1cd44fb5440e753 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/8b6aabe042a54623b1cd44fb5440e753 2023-03-20 10:58:08,917 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/8b6aabe042a54623b1cd44fb5440e753, entries=200, sequenceid=352, filesize=32.3 K 2023-03-20 10:58:08,917 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/B/225de2be28a3471381649b5df8ef51ac as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/225de2be28a3471381649b5df8ef51ac 2023-03-20 10:58:08,920 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/225de2be28a3471381649b5df8ef51ac, entries=150, sequenceid=352, filesize=11.9 K 2023-03-20 10:58:08,920 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/C/4c2ccb73beeb4b0780bd2561ab2c58b8 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/4c2ccb73beeb4b0780bd2561ab2c58b8 2023-03-20 10:58:08,923 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/4c2ccb73beeb4b0780bd2561ab2c58b8, entries=150, sequenceid=352, filesize=11.9 K 2023-03-20 10:58:08,923 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for ee10ffd90d917f64b63db6b593dd0991 in 480ms, sequenceid=352, compaction requested=true 2023-03-20 10:58:08,923 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ee10ffd90d917f64b63db6b593dd0991: 2023-03-20 10:58:08,924 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:58:08,924 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 10:58:08,924 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 10:58:08,924 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:58:08,924 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:58:08,925 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 104858 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 10:58:08,925 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1912): ee10ffd90d917f64b63db6b593dd0991/A is initiating minor compaction (all files) 2023-03-20 10:58:08,925 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ee10ffd90d917f64b63db6b593dd0991/A in TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:08,925 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/8a7825e93ad940298e0e342d22ebba20, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/a14207d61cfd46e5ab8cef7cfc72aa59, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/644f36b2ec244115a01a09d35f4d494d, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/8b6aabe042a54623b1cd44fb5440e753] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp, totalSize=102.4 K 2023-03-20 10:58:08,925 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47016 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 10:58:08,925 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): ee10ffd90d917f64b63db6b593dd0991/B is initiating minor compaction (all files) 2023-03-20 10:58:08,925 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ee10ffd90d917f64b63db6b593dd0991/B in TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:08,925 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/1497d0653d3b4923b056212d696b8940, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/b54a5a0d03c34499ae1e1b7d2898b3c6, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/a489a9fe66bf443794dce0a1e72eaaea, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/225de2be28a3471381649b5df8ef51ac] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp, totalSize=45.9 K 2023-03-20 10:58:08,926 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 8a7825e93ad940298e0e342d22ebba20, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=302, earliestPutTs=1679309884728 2023-03-20 10:58:08,926 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 1497d0653d3b4923b056212d696b8940, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=302, earliestPutTs=1679309884728 2023-03-20 10:58:08,926 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting a14207d61cfd46e5ab8cef7cfc72aa59, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=315, earliestPutTs=1679309884904 2023-03-20 10:58:08,926 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting b54a5a0d03c34499ae1e1b7d2898b3c6, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=315, earliestPutTs=1679309884904 2023-03-20 10:58:08,926 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 644f36b2ec244115a01a09d35f4d494d, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=337, earliestPutTs=1679309885961 2023-03-20 10:58:08,927 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting a489a9fe66bf443794dce0a1e72eaaea, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=337, earliestPutTs=1679309885961 2023-03-20 10:58:08,927 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 8b6aabe042a54623b1cd44fb5440e753, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1679309888121 2023-03-20 10:58:08,927 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 225de2be28a3471381649b5df8ef51ac, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1679309888121 2023-03-20 10:58:08,956 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): ee10ffd90d917f64b63db6b593dd0991#B#compaction#371 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:58:08,992 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] throttle.PressureAwareThroughputController(145): ee10ffd90d917f64b63db6b593dd0991#A#compaction#372 average throughput is 0.66 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:58:08,995 DEBUG [Thread-2246] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:08,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:08,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:09,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:09,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:09,026 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:09,026 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:09,026 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:09,026 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:09,028 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:09,028 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:09,028 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:09,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:09,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:09,029 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:09,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:09,029 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:09,029 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:09,029 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:09,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:09,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:09,030 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:09,033 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:09,033 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:09,033 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:09,033 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:09,033 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:09,042 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:09,042 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:09,042 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:09,042 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. started... 2023-03-20 10:58:09,042 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ee10ffd90d917f64b63db6b593dd0991 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-20 10:58:09,043 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee10ffd90d917f64b63db6b593dd0991, store=A 2023-03-20 10:58:09,043 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:09,043 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee10ffd90d917f64b63db6b593dd0991, store=B 2023-03-20 10:58:09,043 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:09,043 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee10ffd90d917f64b63db6b593dd0991, store=C 2023-03-20 10:58:09,043 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:09,046 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/B/839a7687b8144ef7b3029bd4f20a6bc7 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/839a7687b8144ef7b3029bd4f20a6bc7 2023-03-20 10:58:09,051 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ee10ffd90d917f64b63db6b593dd0991/B of ee10ffd90d917f64b63db6b593dd0991 into 839a7687b8144ef7b3029bd4f20a6bc7(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:58:09,051 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ee10ffd90d917f64b63db6b593dd0991: 2023-03-20 10:58:09,051 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991., storeName=ee10ffd90d917f64b63db6b593dd0991/B, priority=12, startTime=1679309888924; duration=0sec 2023-03-20 10:58:09,051 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:58:09,051 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 10:58:09,052 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47016 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 10:58:09,052 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): ee10ffd90d917f64b63db6b593dd0991/C is initiating minor compaction (all files) 2023-03-20 10:58:09,052 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ee10ffd90d917f64b63db6b593dd0991/C in TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:09,052 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/61400ef950584afbad889a7bba056380, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/e47eae7095be44658d743e29ca81b49e, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/f1d93ff160294bec977c9b7ba8a2c688, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/4c2ccb73beeb4b0780bd2561ab2c58b8] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp, totalSize=45.9 K 2023-03-20 10:58:09,053 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 61400ef950584afbad889a7bba056380, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=302, earliestPutTs=1679309884728 2023-03-20 10:58:09,053 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting e47eae7095be44658d743e29ca81b49e, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=315, earliestPutTs=1679309884904 2023-03-20 10:58:09,054 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting f1d93ff160294bec977c9b7ba8a2c688, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=337, earliestPutTs=1679309885961 2023-03-20 10:58:09,054 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 4c2ccb73beeb4b0780bd2561ab2c58b8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1679309888121 2023-03-20 10:58:09,060 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): ee10ffd90d917f64b63db6b593dd0991#C#compaction#374 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:58:09,064 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. as already flushing 2023-03-20 10:58:09,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on ee10ffd90d917f64b63db6b593dd0991 2023-03-20 10:58:09,109 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:09,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43478 deadline: 1679309949100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:09,110 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:09,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43494 deadline: 1679309949101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:09,110 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:09,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43530 deadline: 1679309949101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:09,120 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:09,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43528 deadline: 1679309949109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:09,120 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:09,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43492 deadline: 1679309949110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:09,213 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:09,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43478 deadline: 1679309949210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:09,216 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:09,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43494 deadline: 1679309949211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:09,216 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:09,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43530 deadline: 1679309949211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:09,226 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:09,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43528 deadline: 1679309949222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:09,226 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:09,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43492 deadline: 1679309949223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:09,346 DEBUG [Thread-2255] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x143a1290 to 127.0.0.1:60226 2023-03-20 10:58:09,346 DEBUG [Thread-2255] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 10:58:09,348 DEBUG [Thread-2247] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x36a9ece3 to 127.0.0.1:60226 2023-03-20 10:58:09,348 DEBUG [Thread-2247] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 10:58:09,352 DEBUG [Thread-2251] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x14d061d2 to 127.0.0.1:60226 2023-03-20 10:58:09,352 DEBUG [Thread-2251] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 10:58:09,353 DEBUG [Thread-2253] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x228ff1a4 to 127.0.0.1:60226 2023-03-20 10:58:09,353 DEBUG [Thread-2253] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 10:58:09,355 DEBUG [Thread-2249] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2ded66ad to 127.0.0.1:60226 2023-03-20 10:58:09,355 DEBUG [Thread-2249] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 10:58:09,414 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:09,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43478 deadline: 1679309949414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:09,417 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:09,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43530 deadline: 1679309949417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:09,418 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:09,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43494 deadline: 1679309949418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:09,427 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:09,427 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:09,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43492 deadline: 1679309949427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:09,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43528 deadline: 1679309949427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:09,467 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/A/855a048656a9474c9ab0fe2f780cfb11 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/855a048656a9474c9ab0fe2f780cfb11 2023-03-20 10:58:09,470 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230320edac907eafe44248bf3b00f55497ca62 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320edac907eafe44248bf3b00f55497ca62 2023-03-20 10:58:09,471 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ee10ffd90d917f64b63db6b593dd0991/A of ee10ffd90d917f64b63db6b593dd0991 into 855a048656a9474c9ab0fe2f780cfb11(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:58:09,471 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ee10ffd90d917f64b63db6b593dd0991: 2023-03-20 10:58:09,471 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991., storeName=ee10ffd90d917f64b63db6b593dd0991/A, priority=12, startTime=1679309888924; duration=0sec 2023-03-20 10:58:09,471 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:58:09,475 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/C/0caedc94693a494296e3758d15519bf6 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/0caedc94693a494296e3758d15519bf6 2023-03-20 10:58:09,478 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ee10ffd90d917f64b63db6b593dd0991/C of ee10ffd90d917f64b63db6b593dd0991 into 0caedc94693a494296e3758d15519bf6(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:58:09,478 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ee10ffd90d917f64b63db6b593dd0991: 2023-03-20 10:58:09,478 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991., storeName=ee10ffd90d917f64b63db6b593dd0991/C, priority=12, startTime=1679309888924; duration=0sec 2023-03-20 10:58:09,478 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:58:09,716 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:09,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43478 deadline: 1679309949716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:09,719 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:09,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43530 deadline: 1679309949719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:09,720 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:09,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43494 deadline: 1679309949720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:09,728 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:09,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43528 deadline: 1679309949728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:09,730 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:09,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43492 deadline: 1679309949730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:09,875 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=369, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/A/1a263f7493904fdcab77397f3732fb1e 2023-03-20 10:58:10,220 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:10,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43478 deadline: 1679309950220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:10,222 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:10,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43530 deadline: 1679309950222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:10,222 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:10,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43494 deadline: 1679309950222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:10,232 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:10,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43528 deadline: 1679309950232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:10,234 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:10,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:43492 deadline: 1679309950234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:10,287 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=369 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/B/7869117d145e4355832fb8bad2d12144 2023-03-20 10:58:10,300 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=369 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/C/f29b3c76e4284f1cac61d68f3e663c32 2023-03-20 10:58:10,304 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/A/1a263f7493904fdcab77397f3732fb1e as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/1a263f7493904fdcab77397f3732fb1e 2023-03-20 10:58:10,310 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/1a263f7493904fdcab77397f3732fb1e, entries=150, sequenceid=369, filesize=25.4 K 2023-03-20 10:58:10,311 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/B/7869117d145e4355832fb8bad2d12144 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/7869117d145e4355832fb8bad2d12144 2023-03-20 10:58:10,316 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/7869117d145e4355832fb8bad2d12144, entries=150, sequenceid=369, filesize=11.9 K 2023-03-20 10:58:10,316 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/C/f29b3c76e4284f1cac61d68f3e663c32 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/f29b3c76e4284f1cac61d68f3e663c32 2023-03-20 10:58:10,320 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/f29b3c76e4284f1cac61d68f3e663c32, entries=150, sequenceid=369, filesize=11.9 K 2023-03-20 10:58:10,320 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=87.22 KB/89310 for ee10ffd90d917f64b63db6b593dd0991 in 1278ms, sequenceid=369, compaction requested=false 2023-03-20 10:58:10,320 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ee10ffd90d917f64b63db6b593dd0991: 2023-03-20 10:58:10,320 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:10,320 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:10,320 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:10,320 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:10,320 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:10,320 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:10,321 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:10,321 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:10,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:10,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:10,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:10,322 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:10,322 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:10,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:10,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:10,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:10,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:10,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:10,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:10,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e18571f[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:10,323 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:10,323 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:10,323 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:10,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:10,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:10,324 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:10,324 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:10,324 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:10,324 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:10,324 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:10,324 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:10,324 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:10,324 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:10,324 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:10,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:10,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:10,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:10,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:10,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:10,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:10,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:10,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:10,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:10,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:10,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:10,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:10,326 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:10,326 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:10,326 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:10,326 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:10,326 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:10,327 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:10,327 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:10,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:10,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:10,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:10,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:10,327 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:10,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:10,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:10,328 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:10,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:10,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:10,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:10,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:10,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:10,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:10,329 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:10,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:10,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:10,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:10,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:10,332 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:10,332 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:10,332 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:10,332 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:10,332 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:10,332 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:10,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:10,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:10,332 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:10,332 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:10,332 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:10,332 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:10,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:10,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:10,332 DEBUG [Thread-2246] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:10,332 DEBUG [Thread-2246] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:10,332 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:10,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:10,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:10,432 DEBUG [Thread-2246] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:10,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:11,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on ee10ffd90d917f64b63db6b593dd0991 2023-03-20 10:58:11,228 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ee10ffd90d917f64b63db6b593dd0991 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-20 10:58:11,229 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee10ffd90d917f64b63db6b593dd0991, store=A 2023-03-20 10:58:11,229 DEBUG [Thread-2238] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x57b6628e to 127.0.0.1:60226 2023-03-20 10:58:11,229 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:11,229 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee10ffd90d917f64b63db6b593dd0991, store=B 2023-03-20 10:58:11,229 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:11,229 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee10ffd90d917f64b63db6b593dd0991, store=C 2023-03-20 10:58:11,230 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:11,229 DEBUG [Thread-2238] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 10:58:11,232 DEBUG [Thread-2236] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x74a355de to 127.0.0.1:60226 2023-03-20 10:58:11,232 DEBUG [Thread-2236] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 10:58:11,233 DEBUG [Thread-2240] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5da5030e to 127.0.0.1:60226 2023-03-20 10:58:11,234 DEBUG [Thread-2240] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 10:58:11,235 DEBUG [Thread-2242] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x74d6bf18 to 127.0.0.1:60226 2023-03-20 10:58:11,235 DEBUG [Thread-2242] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 10:58:11,238 DEBUG [Thread-2244] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x060c5cba to 127.0.0.1:60226 2023-03-20 10:58:11,238 DEBUG [Thread-2244] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 10:58:11,238 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-20 10:58:11,239 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(394): wrote 42 2023-03-20 10:58:11,239 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(394): wrote 63 2023-03-20 10:58:11,239 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(394): wrote 44 2023-03-20 10:58:11,239 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(394): wrote 80 2023-03-20 10:58:11,239 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(394): wrote 77 2023-03-20 10:58:11,239 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-20 10:58:11,239 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-20 10:58:11,239 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(402): scanned 1584 2023-03-20 10:58:11,239 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(403): verified 4752 rows 2023-03-20 10:58:11,239 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(402): scanned 1588 2023-03-20 10:58:11,239 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(403): verified 4764 rows 2023-03-20 10:58:11,239 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(402): scanned 1586 2023-03-20 10:58:11,239 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(403): verified 4758 rows 2023-03-20 10:58:11,240 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(402): scanned 1580 2023-03-20 10:58:11,240 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(403): verified 4739 rows 2023-03-20 10:58:11,240 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(402): scanned 1580 2023-03-20 10:58:11,240 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(403): verified 4740 rows 2023-03-20 10:58:11,240 INFO [Listener at localhost.localdomain/43509] client.ConnectionImplementation(1959): Closing master protocol: MasterService 2023-03-20 10:58:11,240 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7d243e46 to 127.0.0.1:60226 2023-03-20 10:58:11,240 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 10:58:11,241 INFO [Listener at localhost.localdomain/43509] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-20 10:58:11,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-03-20 10:58:11,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure2.ProcedureExecutor(1029): Stored pid=43, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-20 10:58:11,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-20 10:58:11,248 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679309891248"}]},"ts":"1679309891248"} 2023-03-20 10:58:11,249 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-20 10:58:11,253 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-20 10:58:11,253 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ee10ffd90d917f64b63db6b593dd0991, UNASSIGN}] 2023-03-20 10:58:11,255 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ee10ffd90d917f64b63db6b593dd0991, UNASSIGN 2023-03-20 10:58:11,255 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=ee10ffd90d917f64b63db6b593dd0991, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:11,255 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679309891255"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679309891255"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679309891255"}]},"ts":"1679309891255"} 2023-03-20 10:58:11,256 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE; CloseRegionProcedure ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062}] 2023-03-20 10:58:11,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-20 10:58:11,408 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close ee10ffd90d917f64b63db6b593dd0991 2023-03-20 10:58:11,408 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing ee10ffd90d917f64b63db6b593dd0991, disabling compactions & flushes 2023-03-20 10:58:11,408 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:11,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-20 10:58:11,654 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230320e0a98ca6539c451a9b236d2ad8cfb0ef to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320e0a98ca6539c451a9b236d2ad8cfb0ef 2023-03-20 10:58:11,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-20 10:58:12,066 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=389, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/A/e15ee610f9fa4511b5507187ceac612d 2023-03-20 10:58:12,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-20 10:58:12,474 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=389 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/B/563897f8c26942959f2d540b62f1163a 2023-03-20 10:58:12,882 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=389 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/C/1dfb10aa813041be98c9646f9a341d28 2023-03-20 10:58:12,885 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/A/e15ee610f9fa4511b5507187ceac612d as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/e15ee610f9fa4511b5507187ceac612d 2023-03-20 10:58:12,889 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/e15ee610f9fa4511b5507187ceac612d, entries=150, sequenceid=389, filesize=25.4 K 2023-03-20 10:58:12,889 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/B/563897f8c26942959f2d540b62f1163a as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/563897f8c26942959f2d540b62f1163a 2023-03-20 10:58:12,894 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/563897f8c26942959f2d540b62f1163a, entries=150, sequenceid=389, filesize=11.9 K 2023-03-20 10:58:12,894 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/C/1dfb10aa813041be98c9646f9a341d28 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/1dfb10aa813041be98c9646f9a341d28 2023-03-20 10:58:12,899 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/1dfb10aa813041be98c9646f9a341d28, entries=150, sequenceid=389, filesize=11.9 K 2023-03-20 10:58:12,900 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=26.84 KB/27480 for ee10ffd90d917f64b63db6b593dd0991 in 1672ms, sequenceid=389, compaction requested=true 2023-03-20 10:58:12,901 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ee10ffd90d917f64b63db6b593dd0991: 2023-03-20 10:58:12,901 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:12,901 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:58:12,901 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:12,901 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. because compaction request was cancelled 2023-03-20 10:58:12,901 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. after waiting 0 ms 2023-03-20 10:58:12,901 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. because compaction request was cancelled 2023-03-20 10:58:12,901 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:58:12,901 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:12,901 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:58:12,901 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing ee10ffd90d917f64b63db6b593dd0991 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-20 10:58:12,901 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. because compaction request was cancelled 2023-03-20 10:58:12,901 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee10ffd90d917f64b63db6b593dd0991, store=A 2023-03-20 10:58:12,901 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:12,901 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee10ffd90d917f64b63db6b593dd0991, store=B 2023-03-20 10:58:12,901 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:12,902 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee10ffd90d917f64b63db6b593dd0991, store=C 2023-03-20 10:58:12,902 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:12,913 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032043336ede2f874ae5b31c43b714f063ae to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032043336ede2f874ae5b31c43b714f063ae 2023-03-20 10:58:13,320 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=396, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/A/ae8177ef8701426cb87fd8d9c5683af8 2023-03-20 10:58:13,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-20 10:58:13,731 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=396 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/B/9bdb0aabf45d43dd81eb5eb2dbf85a2b 2023-03-20 10:58:14,142 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=396 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/C/328d1e2d13a14a7a94af35a3b7068511 2023-03-20 10:58:14,145 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/A/ae8177ef8701426cb87fd8d9c5683af8 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/ae8177ef8701426cb87fd8d9c5683af8 2023-03-20 10:58:14,148 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/ae8177ef8701426cb87fd8d9c5683af8, entries=150, sequenceid=396, filesize=25.4 K 2023-03-20 10:58:14,148 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/B/9bdb0aabf45d43dd81eb5eb2dbf85a2b as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/9bdb0aabf45d43dd81eb5eb2dbf85a2b 2023-03-20 10:58:14,151 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/9bdb0aabf45d43dd81eb5eb2dbf85a2b, entries=150, sequenceid=396, filesize=11.9 K 2023-03-20 10:58:14,152 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/.tmp/C/328d1e2d13a14a7a94af35a3b7068511 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/328d1e2d13a14a7a94af35a3b7068511 2023-03-20 10:58:14,156 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/328d1e2d13a14a7a94af35a3b7068511, entries=150, sequenceid=396, filesize=11.9 K 2023-03-20 10:58:14,157 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for ee10ffd90d917f64b63db6b593dd0991 in 1256ms, sequenceid=396, compaction requested=true 2023-03-20 10:58:14,178 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/0602a43a94cd4cdf880607998644fb04, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/1decfdfeb5f545388e5e811c1cb52553, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/82c0e3e07fd54edab8508e6d284270a6, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/6b2131cad89a471cb9149d29551e91e5, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/f46475f61d6b46bb82b447f2cb0a96f8, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/14a11b7bfca34fecbfe24856b5d70122, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/c1d27d0a757442b1b9be344d34fe2819, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/d496e90bb995417c9b71d15bafb8fc99, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/8f8f523d000247678ba5c8e3056b08fc, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/026e32c9064f48e792c69f902a7c5ca6, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/74f1892379b74f1580b12d64ef2aa019, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/c2b467af561c4214b43b705fe3d6e091, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/88d408d50c91415ba1c4db6cdfc749c1, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/45da4837dcb74f89abc1e30f6855f2a0, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/26340732b60145d1af43001b00da6166, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/98228da5c6584de4b6fffc4e081c9fbf, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/f898ad23c6b64a669103478af37b965f, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/3ab8e1a10ec84e5795702b4e2b0e11ba, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/fb8b8aca2e474df0ab259b7a3658423e, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/d6de0dc7ad8c431e9ddc0d4a942fd140, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/8a7825e93ad940298e0e342d22ebba20, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/a14207d61cfd46e5ab8cef7cfc72aa59, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/644f36b2ec244115a01a09d35f4d494d, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/8b6aabe042a54623b1cd44fb5440e753] to archive 2023-03-20 10:58:14,179 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-20 10:58:14,180 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/0602a43a94cd4cdf880607998644fb04 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/0602a43a94cd4cdf880607998644fb04 2023-03-20 10:58:14,181 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/1decfdfeb5f545388e5e811c1cb52553 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/1decfdfeb5f545388e5e811c1cb52553 2023-03-20 10:58:14,182 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/82c0e3e07fd54edab8508e6d284270a6 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/82c0e3e07fd54edab8508e6d284270a6 2023-03-20 10:58:14,183 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/6b2131cad89a471cb9149d29551e91e5 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/6b2131cad89a471cb9149d29551e91e5 2023-03-20 10:58:14,184 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/f46475f61d6b46bb82b447f2cb0a96f8 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/f46475f61d6b46bb82b447f2cb0a96f8 2023-03-20 10:58:14,185 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/14a11b7bfca34fecbfe24856b5d70122 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/14a11b7bfca34fecbfe24856b5d70122 2023-03-20 10:58:14,186 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/c1d27d0a757442b1b9be344d34fe2819 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/c1d27d0a757442b1b9be344d34fe2819 2023-03-20 10:58:14,187 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/d496e90bb995417c9b71d15bafb8fc99 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/d496e90bb995417c9b71d15bafb8fc99 2023-03-20 10:58:14,189 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/8f8f523d000247678ba5c8e3056b08fc to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/8f8f523d000247678ba5c8e3056b08fc 2023-03-20 10:58:14,190 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/026e32c9064f48e792c69f902a7c5ca6 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/026e32c9064f48e792c69f902a7c5ca6 2023-03-20 10:58:14,190 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/74f1892379b74f1580b12d64ef2aa019 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/74f1892379b74f1580b12d64ef2aa019 2023-03-20 10:58:14,192 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/c2b467af561c4214b43b705fe3d6e091 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/c2b467af561c4214b43b705fe3d6e091 2023-03-20 10:58:14,193 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/88d408d50c91415ba1c4db6cdfc749c1 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/88d408d50c91415ba1c4db6cdfc749c1 2023-03-20 10:58:14,194 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/45da4837dcb74f89abc1e30f6855f2a0 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/45da4837dcb74f89abc1e30f6855f2a0 2023-03-20 10:58:14,195 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/26340732b60145d1af43001b00da6166 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/26340732b60145d1af43001b00da6166 2023-03-20 10:58:14,196 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/98228da5c6584de4b6fffc4e081c9fbf to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/98228da5c6584de4b6fffc4e081c9fbf 2023-03-20 10:58:14,197 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/f898ad23c6b64a669103478af37b965f to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/f898ad23c6b64a669103478af37b965f 2023-03-20 10:58:14,198 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/3ab8e1a10ec84e5795702b4e2b0e11ba to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/3ab8e1a10ec84e5795702b4e2b0e11ba 2023-03-20 10:58:14,199 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/fb8b8aca2e474df0ab259b7a3658423e to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/fb8b8aca2e474df0ab259b7a3658423e 2023-03-20 10:58:14,200 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/d6de0dc7ad8c431e9ddc0d4a942fd140 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/d6de0dc7ad8c431e9ddc0d4a942fd140 2023-03-20 10:58:14,200 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/8a7825e93ad940298e0e342d22ebba20 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/8a7825e93ad940298e0e342d22ebba20 2023-03-20 10:58:14,201 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/a14207d61cfd46e5ab8cef7cfc72aa59 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/a14207d61cfd46e5ab8cef7cfc72aa59 2023-03-20 10:58:14,202 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/644f36b2ec244115a01a09d35f4d494d to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/644f36b2ec244115a01a09d35f4d494d 2023-03-20 10:58:14,203 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/8b6aabe042a54623b1cd44fb5440e753 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/8b6aabe042a54623b1cd44fb5440e753 2023-03-20 10:58:14,205 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/8167b7abdc364b828f2f5a16e552b1b5, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/dbb1dfcafa414400804ae42811f6cdc3, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/176eb5c60819464f876e82e90e3ba7a4, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/d54def83c41744209cde5c712c33fee4, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/369da498a1d949f1a4305a3b881de17d, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/ab9fe79b9e2d438db36565392a48e7ea, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/c863e9d7ff39436ebfdb7610eb0c0040, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/d81acef791404fc09c434ff0ea87e8f0, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/957a683fe95c49a8949b86da2763ca16, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/6708edf3beec490ab0550473db651135, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/68a019c95427498c9c7777d7b863052c, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/7bf0ba77a62e46199b553fdf51c94dac, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/90458d0d168c42e6be839aa6ceeb1bc3, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/c44c44475b51498b9cac38c24c7b1840, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/282030eb96ea423e8a055a9b262155b0, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/3e5c064714b24214923ae32a78d0d86c, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/a084c7d4e8464f1a8f5feaa7d12465b4, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/dced5faefb7e47ff9f6a410cdeb28b0a, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/0c9aa65593ab42e5bdde7580a43a6679, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/0467d0e9f001484eb2a0527a3de86060, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/1497d0653d3b4923b056212d696b8940, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/dfaf99f8655542d28dbaec0be6d52bac, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/b54a5a0d03c34499ae1e1b7d2898b3c6, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/a489a9fe66bf443794dce0a1e72eaaea, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/225de2be28a3471381649b5df8ef51ac] to archive 2023-03-20 10:58:14,206 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-20 10:58:14,207 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/8167b7abdc364b828f2f5a16e552b1b5 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/8167b7abdc364b828f2f5a16e552b1b5 2023-03-20 10:58:14,208 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/dbb1dfcafa414400804ae42811f6cdc3 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/dbb1dfcafa414400804ae42811f6cdc3 2023-03-20 10:58:14,208 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/176eb5c60819464f876e82e90e3ba7a4 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/176eb5c60819464f876e82e90e3ba7a4 2023-03-20 10:58:14,209 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/d54def83c41744209cde5c712c33fee4 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/d54def83c41744209cde5c712c33fee4 2023-03-20 10:58:14,210 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/369da498a1d949f1a4305a3b881de17d to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/369da498a1d949f1a4305a3b881de17d 2023-03-20 10:58:14,210 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/ab9fe79b9e2d438db36565392a48e7ea to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/ab9fe79b9e2d438db36565392a48e7ea 2023-03-20 10:58:14,211 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/c863e9d7ff39436ebfdb7610eb0c0040 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/c863e9d7ff39436ebfdb7610eb0c0040 2023-03-20 10:58:14,212 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/d81acef791404fc09c434ff0ea87e8f0 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/d81acef791404fc09c434ff0ea87e8f0 2023-03-20 10:58:14,213 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/957a683fe95c49a8949b86da2763ca16 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/957a683fe95c49a8949b86da2763ca16 2023-03-20 10:58:14,213 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/6708edf3beec490ab0550473db651135 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/6708edf3beec490ab0550473db651135 2023-03-20 10:58:14,214 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/68a019c95427498c9c7777d7b863052c to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/68a019c95427498c9c7777d7b863052c 2023-03-20 10:58:14,215 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/7bf0ba77a62e46199b553fdf51c94dac to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/7bf0ba77a62e46199b553fdf51c94dac 2023-03-20 10:58:14,216 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/90458d0d168c42e6be839aa6ceeb1bc3 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/90458d0d168c42e6be839aa6ceeb1bc3 2023-03-20 10:58:14,216 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/c44c44475b51498b9cac38c24c7b1840 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/c44c44475b51498b9cac38c24c7b1840 2023-03-20 10:58:14,217 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/282030eb96ea423e8a055a9b262155b0 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/282030eb96ea423e8a055a9b262155b0 2023-03-20 10:58:14,218 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/3e5c064714b24214923ae32a78d0d86c to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/3e5c064714b24214923ae32a78d0d86c 2023-03-20 10:58:14,219 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/a084c7d4e8464f1a8f5feaa7d12465b4 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/a084c7d4e8464f1a8f5feaa7d12465b4 2023-03-20 10:58:14,220 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/dced5faefb7e47ff9f6a410cdeb28b0a to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/dced5faefb7e47ff9f6a410cdeb28b0a 2023-03-20 10:58:14,221 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/0c9aa65593ab42e5bdde7580a43a6679 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/0c9aa65593ab42e5bdde7580a43a6679 2023-03-20 10:58:14,222 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/0467d0e9f001484eb2a0527a3de86060 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/0467d0e9f001484eb2a0527a3de86060 2023-03-20 10:58:14,223 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/1497d0653d3b4923b056212d696b8940 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/1497d0653d3b4923b056212d696b8940 2023-03-20 10:58:14,224 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/dfaf99f8655542d28dbaec0be6d52bac to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/dfaf99f8655542d28dbaec0be6d52bac 2023-03-20 10:58:14,225 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/b54a5a0d03c34499ae1e1b7d2898b3c6 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/b54a5a0d03c34499ae1e1b7d2898b3c6 2023-03-20 10:58:14,226 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/a489a9fe66bf443794dce0a1e72eaaea to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/a489a9fe66bf443794dce0a1e72eaaea 2023-03-20 10:58:14,227 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/225de2be28a3471381649b5df8ef51ac to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/225de2be28a3471381649b5df8ef51ac 2023-03-20 10:58:14,231 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/00f4ffb13fb54e29b53d38a691eadd1c, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/fe6869de19cc482097839196c5a0695b, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/dd2696344d8442a79ac6f57f49b6720e, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/aaf998f0e8cd4c3d8b2f1d13bec40a3f, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/edee06c383124e21bb308afaf7bbf239, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/e48c013af12f4b559b28431c77989c96, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/de865513902140438491cb469b37f997, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/08c958c21ecb4751a10a92ffdbe1feaa, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/e6d6072d1f194b01858b0258e058c3cf, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/a19007863ea047cba240f46626498f7b, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/5c87af7b44664630b6c7848509329ea0, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/d8c9d6bdad324d439706f0cd4e57d6b0, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/3ed679e38c8c47d79b39dd77b5989c68, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/643397ed8be644b7b4c68ba9a8328a87, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/1745645435e14d108c602095c57ca9cf, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/4a16b1810ced4f6f82b05b775ea6f0c1, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/c0f6ef5209844c2c9a8dab31926d3764, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/9730fad3c5d4470db0df5aa740d2a3a9, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/7cd648fa1dd645078b2e5ccfb65fcdaf, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/a69e6341380344a1b13fe9e69e302387, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/61400ef950584afbad889a7bba056380, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/07c6d6c76aba4853b007d6e3868a0ed4, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/e47eae7095be44658d743e29ca81b49e, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/f1d93ff160294bec977c9b7ba8a2c688, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/4c2ccb73beeb4b0780bd2561ab2c58b8] to archive 2023-03-20 10:58:14,232 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-20 10:58:14,233 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/00f4ffb13fb54e29b53d38a691eadd1c to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/00f4ffb13fb54e29b53d38a691eadd1c 2023-03-20 10:58:14,235 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/fe6869de19cc482097839196c5a0695b to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/fe6869de19cc482097839196c5a0695b 2023-03-20 10:58:14,236 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/dd2696344d8442a79ac6f57f49b6720e to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/dd2696344d8442a79ac6f57f49b6720e 2023-03-20 10:58:14,237 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/aaf998f0e8cd4c3d8b2f1d13bec40a3f to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/aaf998f0e8cd4c3d8b2f1d13bec40a3f 2023-03-20 10:58:14,238 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/edee06c383124e21bb308afaf7bbf239 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/edee06c383124e21bb308afaf7bbf239 2023-03-20 10:58:14,239 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/e48c013af12f4b559b28431c77989c96 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/e48c013af12f4b559b28431c77989c96 2023-03-20 10:58:14,240 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/de865513902140438491cb469b37f997 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/de865513902140438491cb469b37f997 2023-03-20 10:58:14,241 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/08c958c21ecb4751a10a92ffdbe1feaa to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/08c958c21ecb4751a10a92ffdbe1feaa 2023-03-20 10:58:14,242 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/e6d6072d1f194b01858b0258e058c3cf to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/e6d6072d1f194b01858b0258e058c3cf 2023-03-20 10:58:14,243 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/a19007863ea047cba240f46626498f7b to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/a19007863ea047cba240f46626498f7b 2023-03-20 10:58:14,244 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/5c87af7b44664630b6c7848509329ea0 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/5c87af7b44664630b6c7848509329ea0 2023-03-20 10:58:14,245 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/d8c9d6bdad324d439706f0cd4e57d6b0 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/d8c9d6bdad324d439706f0cd4e57d6b0 2023-03-20 10:58:14,246 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/3ed679e38c8c47d79b39dd77b5989c68 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/3ed679e38c8c47d79b39dd77b5989c68 2023-03-20 10:58:14,247 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/643397ed8be644b7b4c68ba9a8328a87 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/643397ed8be644b7b4c68ba9a8328a87 2023-03-20 10:58:14,247 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/1745645435e14d108c602095c57ca9cf to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/1745645435e14d108c602095c57ca9cf 2023-03-20 10:58:14,248 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/4a16b1810ced4f6f82b05b775ea6f0c1 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/4a16b1810ced4f6f82b05b775ea6f0c1 2023-03-20 10:58:14,249 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/c0f6ef5209844c2c9a8dab31926d3764 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/c0f6ef5209844c2c9a8dab31926d3764 2023-03-20 10:58:14,250 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/9730fad3c5d4470db0df5aa740d2a3a9 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/9730fad3c5d4470db0df5aa740d2a3a9 2023-03-20 10:58:14,251 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/7cd648fa1dd645078b2e5ccfb65fcdaf to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/7cd648fa1dd645078b2e5ccfb65fcdaf 2023-03-20 10:58:14,251 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/a69e6341380344a1b13fe9e69e302387 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/a69e6341380344a1b13fe9e69e302387 2023-03-20 10:58:14,252 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/61400ef950584afbad889a7bba056380 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/61400ef950584afbad889a7bba056380 2023-03-20 10:58:14,253 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/07c6d6c76aba4853b007d6e3868a0ed4 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/07c6d6c76aba4853b007d6e3868a0ed4 2023-03-20 10:58:14,254 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/e47eae7095be44658d743e29ca81b49e to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/e47eae7095be44658d743e29ca81b49e 2023-03-20 10:58:14,254 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/f1d93ff160294bec977c9b7ba8a2c688 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/f1d93ff160294bec977c9b7ba8a2c688 2023-03-20 10:58:14,255 DEBUG [StoreCloser-TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/4c2ccb73beeb4b0780bd2561ab2c58b8 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/4c2ccb73beeb4b0780bd2561ab2c58b8 2023-03-20 10:58:14,258 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/recovered.edits/399.seqid, newMaxSeqId=399, maxSeqId=4 2023-03-20 10:58:14,259 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991. 2023-03-20 10:58:14,259 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for ee10ffd90d917f64b63db6b593dd0991: 2023-03-20 10:58:14,261 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed ee10ffd90d917f64b63db6b593dd0991 2023-03-20 10:58:14,261 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=ee10ffd90d917f64b63db6b593dd0991, regionState=CLOSED 2023-03-20 10:58:14,261 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679309894261"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679309894261"}]},"ts":"1679309894261"} 2023-03-20 10:58:14,263 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=44 2023-03-20 10:58:14,264 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=44, state=SUCCESS; CloseRegionProcedure ee10ffd90d917f64b63db6b593dd0991, server=jenkins-hbase20.apache.org,36041,1679309777062 in 3.0060 sec 2023-03-20 10:58:14,265 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=43 2023-03-20 10:58:14,265 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=43, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=ee10ffd90d917f64b63db6b593dd0991, UNASSIGN in 3.0110 sec 2023-03-20 10:58:14,265 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679309894265"}]},"ts":"1679309894265"} 2023-03-20 10:58:14,266 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-20 10:58:14,267 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-20 10:58:14,269 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=43, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 3.0260 sec 2023-03-20 10:58:15,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-20 10:58:15,355 INFO [Listener at localhost.localdomain/43509] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 43 completed 2023-03-20 10:58:15,357 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-03-20 10:58:15,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure2.ProcedureExecutor(1029): Stored pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-20 10:58:15,361 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-20 10:58:15,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-03-20 10:58:15,362 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=46, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-20 10:58:15,368 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991 2023-03-20 10:58:15,371 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A, FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B, FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C, FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/recovered.edits] 2023-03-20 10:58:15,375 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/1a263f7493904fdcab77397f3732fb1e to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/1a263f7493904fdcab77397f3732fb1e 2023-03-20 10:58:15,376 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/855a048656a9474c9ab0fe2f780cfb11 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/855a048656a9474c9ab0fe2f780cfb11 2023-03-20 10:58:15,376 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/ae8177ef8701426cb87fd8d9c5683af8 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/ae8177ef8701426cb87fd8d9c5683af8 2023-03-20 10:58:15,377 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/e15ee610f9fa4511b5507187ceac612d to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/A/e15ee610f9fa4511b5507187ceac612d 2023-03-20 10:58:15,379 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/563897f8c26942959f2d540b62f1163a to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/563897f8c26942959f2d540b62f1163a 2023-03-20 10:58:15,380 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/7869117d145e4355832fb8bad2d12144 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/7869117d145e4355832fb8bad2d12144 2023-03-20 10:58:15,380 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/839a7687b8144ef7b3029bd4f20a6bc7 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/839a7687b8144ef7b3029bd4f20a6bc7 2023-03-20 10:58:15,381 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/9bdb0aabf45d43dd81eb5eb2dbf85a2b to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/B/9bdb0aabf45d43dd81eb5eb2dbf85a2b 2023-03-20 10:58:15,383 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/0caedc94693a494296e3758d15519bf6 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/0caedc94693a494296e3758d15519bf6 2023-03-20 10:58:15,384 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/1dfb10aa813041be98c9646f9a341d28 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/1dfb10aa813041be98c9646f9a341d28 2023-03-20 10:58:15,385 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/328d1e2d13a14a7a94af35a3b7068511 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/328d1e2d13a14a7a94af35a3b7068511 2023-03-20 10:58:15,386 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/f29b3c76e4284f1cac61d68f3e663c32 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/C/f29b3c76e4284f1cac61d68f3e663c32 2023-03-20 10:58:15,387 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/recovered.edits/399.seqid to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991/recovered.edits/399.seqid 2023-03-20 10:58:15,388 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/ee10ffd90d917f64b63db6b593dd0991 2023-03-20 10:58:15,388 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-20 10:58:15,388 DEBUG [PEWorker-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-20 10:58:15,389 DEBUG [PEWorker-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-03-20 10:58:15,391 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032009570ca0671d421e88b6416d5e3d71dd to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032009570ca0671d421e88b6416d5e3d71dd 2023-03-20 10:58:15,392 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303200b34e45923fd4109975e03b00de8166e to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303200b34e45923fd4109975e03b00de8166e 2023-03-20 10:58:15,393 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303201493c7970db04d1c9aa8d6d9b655db1f to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303201493c7970db04d1c9aa8d6d9b655db1f 2023-03-20 10:58:15,394 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303201f88a48d7e8b4964a484f3fcbc9df7f3 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303201f88a48d7e8b4964a484f3fcbc9df7f3 2023-03-20 10:58:15,394 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303203594524ae8ee458db76da43d2100c840 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303203594524ae8ee458db76da43d2100c840 2023-03-20 10:58:15,395 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032043336ede2f874ae5b31c43b714f063ae to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032043336ede2f874ae5b31c43b714f063ae 2023-03-20 10:58:15,396 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320574f61896b4b495face029905a816368 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320574f61896b4b495face029905a816368 2023-03-20 10:58:15,397 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032057a584a8ac2a485fb356e72a271f3b8b to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032057a584a8ac2a485fb356e72a271f3b8b 2023-03-20 10:58:15,397 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303206e2f118987c04cfaa6cba9f44d943802 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303206e2f118987c04cfaa6cba9f44d943802 2023-03-20 10:58:15,398 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320763bac6158764697be62775ed97f9f0b to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320763bac6158764697be62775ed97f9f0b 2023-03-20 10:58:15,399 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320793e8ec8c7c643d0818ee0c079c6d612 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320793e8ec8c7c643d0818ee0c079c6d612 2023-03-20 10:58:15,400 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320878ddeb6c6d94db8b113d7b401d9d231 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320878ddeb6c6d94db8b113d7b401d9d231 2023-03-20 10:58:15,400 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303208e0ec7b9735949e4b91338fc7f623d18 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303208e0ec7b9735949e4b91338fc7f623d18 2023-03-20 10:58:15,401 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320a857282d19f34afa85a906e5017482c8 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320a857282d19f34afa85a906e5017482c8 2023-03-20 10:58:15,402 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320af5b92c660684fa28326d751f689a2c5 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320af5b92c660684fa28326d751f689a2c5 2023-03-20 10:58:15,403 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320b16b889a7c7142e1b14c50175ea20cbb to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320b16b889a7c7142e1b14c50175ea20cbb 2023-03-20 10:58:15,403 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320d5ad38daba754cb7bb99afda409a970d to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320d5ad38daba754cb7bb99afda409a970d 2023-03-20 10:58:15,404 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320d7b01b2d63f644d9bb51dffb3eece691 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320d7b01b2d63f644d9bb51dffb3eece691 2023-03-20 10:58:15,405 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320dd46ad220d2043bdac1436c307c418e5 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320dd46ad220d2043bdac1436c307c418e5 2023-03-20 10:58:15,406 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320e0a98ca6539c451a9b236d2ad8cfb0ef to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320e0a98ca6539c451a9b236d2ad8cfb0ef 2023-03-20 10:58:15,406 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320e66f3f0652c64d1295e0a6de0a86fe06 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320e66f3f0652c64d1295e0a6de0a86fe06 2023-03-20 10:58:15,407 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320edac907eafe44248bf3b00f55497ca62 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320edac907eafe44248bf3b00f55497ca62 2023-03-20 10:58:15,408 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320fca12e0f419c41019753b9aec808f72b to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320fca12e0f419c41019753b9aec808f72b 2023-03-20 10:58:15,408 DEBUG [PEWorker-3] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-20 10:58:15,410 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=46, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-20 10:58:15,411 WARN [PEWorker-3] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-20 10:58:15,412 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-20 10:58:15,413 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=46, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-20 10:58:15,413 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-20 10:58:15,413 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1679309895413"}]},"ts":"9223372036854775807"} 2023-03-20 10:58:15,415 INFO [PEWorker-3] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-20 10:58:15,415 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => ee10ffd90d917f64b63db6b593dd0991, NAME => 'TestAcidGuarantees,,1679309866252.ee10ffd90d917f64b63db6b593dd0991.', STARTKEY => '', ENDKEY => ''}] 2023-03-20 10:58:15,415 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-20 10:58:15,415 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1679309895415"}]},"ts":"9223372036854775807"} 2023-03-20 10:58:15,416 INFO [PEWorker-3] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-20 10:58:15,417 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(130): Finished pid=46, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-20 10:58:15,417 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=46, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 60 msec 2023-03-20 10:58:15,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-03-20 10:58:15,463 INFO [Listener at localhost.localdomain/43509] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 46 completed 2023-03-20 10:58:15,474 INFO [Listener at localhost.localdomain/43509] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=227 (was 228), OpenFileDescriptor=553 (was 555), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=775 (was 842), ProcessCount=184 (was 184), AvailableMemoryMB=8955 (was 8434) - AvailableMemoryMB LEAK? - 2023-03-20 10:58:15,486 INFO [Listener at localhost.localdomain/43509] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=227, OpenFileDescriptor=553, MaxFileDescriptor=60000, SystemLoadAverage=775, ProcessCount=184, AvailableMemoryMB=8954 2023-03-20 10:58:15,489 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-20 10:58:15,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-20 10:58:15,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure2.ProcedureExecutor(1029): Stored pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-20 10:58:15,492 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-20 10:58:15,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 47 2023-03-20 10:58:15,493 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-20 10:58:15,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-20 10:58:15,494 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01 2023-03-20 10:58:15,495 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01 empty. 2023-03-20 10:58:15,495 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01 2023-03-20 10:58:15,495 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-20 10:58:15,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-20 10:58:15,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-20 10:58:15,903 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-20 10:58:15,905 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 00ef1cabe628ad083ccaa287ccbbfb01, NAME => 'TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp 2023-03-20 10:58:15,913 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-20 10:58:15,913 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 00ef1cabe628ad083ccaa287ccbbfb01, disabling compactions & flushes 2023-03-20 10:58:15,913 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:15,913 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:15,913 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. after waiting 0 ms 2023-03-20 10:58:15,913 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:15,913 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:15,913 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 00ef1cabe628ad083ccaa287ccbbfb01: 2023-03-20 10:58:15,915 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-20 10:58:15,916 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679309895915"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679309895915"}]},"ts":"1679309895915"} 2023-03-20 10:58:15,916 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-20 10:58:15,917 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-20 10:58:15,917 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679309895917"}]},"ts":"1679309895917"} 2023-03-20 10:58:15,918 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-20 10:58:15,919 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=00ef1cabe628ad083ccaa287ccbbfb01, ASSIGN}] 2023-03-20 10:58:15,921 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=00ef1cabe628ad083ccaa287ccbbfb01, ASSIGN 2023-03-20 10:58:15,922 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=00ef1cabe628ad083ccaa287ccbbfb01, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,36041,1679309777062; forceNewPlan=false, retain=false 2023-03-20 10:58:16,073 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=00ef1cabe628ad083ccaa287ccbbfb01, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:16,073 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679309896073"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679309896073"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679309896073"}]},"ts":"1679309896073"} 2023-03-20 10:58:16,074 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=49, ppid=48, state=RUNNABLE; OpenRegionProcedure 00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062}] 2023-03-20 10:58:16,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-20 10:58:16,234 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:16,234 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 00ef1cabe628ad083ccaa287ccbbfb01, NAME => 'TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.', STARTKEY => '', ENDKEY => ''} 2023-03-20 10:58:16,235 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 00ef1cabe628ad083ccaa287ccbbfb01 2023-03-20 10:58:16,235 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-20 10:58:16,235 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 00ef1cabe628ad083ccaa287ccbbfb01 2023-03-20 10:58:16,235 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 00ef1cabe628ad083ccaa287ccbbfb01 2023-03-20 10:58:16,238 INFO [StoreOpener-00ef1cabe628ad083ccaa287ccbbfb01-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 00ef1cabe628ad083ccaa287ccbbfb01 2023-03-20 10:58:16,239 DEBUG [StoreOpener-00ef1cabe628ad083ccaa287ccbbfb01-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A 2023-03-20 10:58:16,239 DEBUG [StoreOpener-00ef1cabe628ad083ccaa287ccbbfb01-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A 2023-03-20 10:58:16,239 INFO [StoreOpener-00ef1cabe628ad083ccaa287ccbbfb01-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-20 10:58:16,240 INFO [StoreOpener-00ef1cabe628ad083ccaa287ccbbfb01-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 00ef1cabe628ad083ccaa287ccbbfb01 columnFamilyName A 2023-03-20 10:58:16,240 INFO [StoreOpener-00ef1cabe628ad083ccaa287ccbbfb01-1] regionserver.HStore(310): Store=00ef1cabe628ad083ccaa287ccbbfb01/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-20 10:58:16,240 INFO [StoreOpener-00ef1cabe628ad083ccaa287ccbbfb01-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 00ef1cabe628ad083ccaa287ccbbfb01 2023-03-20 10:58:16,242 DEBUG [StoreOpener-00ef1cabe628ad083ccaa287ccbbfb01-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B 2023-03-20 10:58:16,242 DEBUG [StoreOpener-00ef1cabe628ad083ccaa287ccbbfb01-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B 2023-03-20 10:58:16,242 INFO [StoreOpener-00ef1cabe628ad083ccaa287ccbbfb01-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-20 10:58:16,242 INFO [StoreOpener-00ef1cabe628ad083ccaa287ccbbfb01-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 00ef1cabe628ad083ccaa287ccbbfb01 columnFamilyName B 2023-03-20 10:58:16,243 INFO [StoreOpener-00ef1cabe628ad083ccaa287ccbbfb01-1] regionserver.HStore(310): Store=00ef1cabe628ad083ccaa287ccbbfb01/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-20 10:58:16,243 INFO [StoreOpener-00ef1cabe628ad083ccaa287ccbbfb01-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 00ef1cabe628ad083ccaa287ccbbfb01 2023-03-20 10:58:16,244 DEBUG [StoreOpener-00ef1cabe628ad083ccaa287ccbbfb01-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C 2023-03-20 10:58:16,244 DEBUG [StoreOpener-00ef1cabe628ad083ccaa287ccbbfb01-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C 2023-03-20 10:58:16,244 INFO [StoreOpener-00ef1cabe628ad083ccaa287ccbbfb01-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-20 10:58:16,244 INFO [StoreOpener-00ef1cabe628ad083ccaa287ccbbfb01-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 00ef1cabe628ad083ccaa287ccbbfb01 columnFamilyName C 2023-03-20 10:58:16,245 INFO [StoreOpener-00ef1cabe628ad083ccaa287ccbbfb01-1] regionserver.HStore(310): Store=00ef1cabe628ad083ccaa287ccbbfb01/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-20 10:58:16,245 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:16,246 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01 2023-03-20 10:58:16,246 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01 2023-03-20 10:58:16,249 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-20 10:58:16,250 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 00ef1cabe628ad083ccaa287ccbbfb01 2023-03-20 10:58:16,252 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-20 10:58:16,252 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 00ef1cabe628ad083ccaa287ccbbfb01; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62040045, jitterRate=-0.07553128898143768}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-20 10:58:16,252 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 00ef1cabe628ad083ccaa287ccbbfb01: 2023-03-20 10:58:16,253 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01., pid=49, masterSystemTime=1679309896226 2023-03-20 10:58:16,254 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:16,254 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:16,254 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=00ef1cabe628ad083ccaa287ccbbfb01, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:16,254 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679309896254"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1679309896254"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679309896254"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679309896254"}]},"ts":"1679309896254"} 2023-03-20 10:58:16,257 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=49, resume processing ppid=48 2023-03-20 10:58:16,257 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=49, ppid=48, state=SUCCESS; OpenRegionProcedure 00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 in 181 msec 2023-03-20 10:58:16,258 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=47 2023-03-20 10:58:16,258 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=47, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=00ef1cabe628ad083ccaa287ccbbfb01, ASSIGN in 338 msec 2023-03-20 10:58:16,259 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-20 10:58:16,259 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679309896259"}]},"ts":"1679309896259"} 2023-03-20 10:58:16,259 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-20 10:58:16,261 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-20 10:58:16,262 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=47, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 772 msec 2023-03-20 10:58:16,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-20 10:58:16,601 INFO [Listener at localhost.localdomain/43509] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 47 completed 2023-03-20 10:58:16,602 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(139): Connect 0x1a885842 to 127.0.0.1:60226 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 10:58:16,611 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@9b9721, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 10:58:16,612 DEBUG [Listener at localhost.localdomain/43509] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 10:58:16,615 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:52130, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 10:58:16,616 DEBUG [Listener at localhost.localdomain/43509] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-20 10:58:16,618 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:51232, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-20 10:58:16,620 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(139): Connect 0x02e8c6f6 to 127.0.0.1:60226 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 10:58:16,626 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5a7ac4a1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 10:58:16,626 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(139): Connect 0x2cb7a788 to 127.0.0.1:60226 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 10:58:16,637 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@682c0b9e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 10:58:16,637 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(139): Connect 0x41ccedcb to 127.0.0.1:60226 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 10:58:16,646 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@50d6899d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 10:58:16,646 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(139): Connect 0x40e6f624 to 127.0.0.1:60226 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 10:58:16,654 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6662e946, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 10:58:16,654 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(139): Connect 0x2a5d7fcf to 127.0.0.1:60226 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 10:58:16,658 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5d7deb76, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 10:58:16,658 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(139): Connect 0x0f4b7807 to 127.0.0.1:60226 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 10:58:16,670 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3d5f04c3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 10:58:16,670 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(139): Connect 0x31fc0e3f to 127.0.0.1:60226 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 10:58:16,678 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6523d347, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 10:58:16,679 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(139): Connect 0x79db57d4 to 127.0.0.1:60226 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 10:58:16,684 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@28072326, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 10:58:16,685 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(139): Connect 0x0e56ea94 to 127.0.0.1:60226 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 10:58:16,690 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2d90797d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 10:58:16,690 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(139): Connect 0x4486b3c7 to 127.0.0.1:60226 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 10:58:16,694 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7d3df389, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 10:58:16,704 DEBUG [hconnection-0x5c8ed8f8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 10:58:16,705 DEBUG [hconnection-0x1e9c3ed2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 10:58:16,707 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:52136, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 10:58:16,707 DEBUG [hconnection-0x2c0a44e0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 10:58:16,708 DEBUG [hconnection-0x19aeba59-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 10:58:16,710 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:52152, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 10:58:16,717 DEBUG [hconnection-0x44c030d6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 10:58:16,717 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:52176, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 10:58:16,717 DEBUG [hconnection-0x702f8032-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 10:58:16,719 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:52164, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 10:58:16,719 DEBUG [hconnection-0x777287ca-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 10:58:16,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:16,720 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:52182, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 10:58:16,724 DEBUG [hconnection-0x509d23ea-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 10:58:16,724 DEBUG [hconnection-0x67c44c26-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 10:58:16,727 DEBUG [hconnection-0x79e895e2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 10:58:16,728 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:52190, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 10:58:16,728 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:52192, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 10:58:16,732 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:52180, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 10:58:16,732 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:52214, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 10:58:16,732 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:52200, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 10:58:16,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 00ef1cabe628ad083ccaa287ccbbfb01 2023-03-20 10:58:16,739 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 00ef1cabe628ad083ccaa287ccbbfb01 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-20 10:58:16,739 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=A 2023-03-20 10:58:16,739 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:16,739 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=B 2023-03-20 10:58:16,739 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:16,739 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=C 2023-03-20 10:58:16,739 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:16,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:16,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:16,742 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:16,742 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:16,742 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:16,742 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:16,742 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:16,743 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:16,743 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:16,743 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:16,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:16,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:16,744 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:16,744 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:16,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:16,744 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:16,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:16,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:16,744 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:16,745 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:16,745 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:16,745 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:16,745 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:16,745 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:16,745 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:16,745 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:16,745 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:16,745 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:16,745 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. as already flushing 2023-03-20 10:58:16,745 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:16,745 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:16,745 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:16,745 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:16,746 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:16,746 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:16,747 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:16,747 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:16,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:16,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:16,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:16,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:16,747 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:16,747 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:16,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:16,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:16,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:16,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:16,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:16,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e680361[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:16,752 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:16,752 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:16,753 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:16,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:16,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:16,753 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:16,753 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:16,753 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:16,753 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:16,753 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:16,754 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:16,754 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:16,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:16,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:16,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:16,754 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:16,754 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:16,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:16,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:16,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:16,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:16,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:16,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:16,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:16,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:16,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:16,756 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:16,756 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:16,756 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:16,756 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:16,757 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:16,758 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:16,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:16,758 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:16,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:16,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:16,758 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:16,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52176 deadline: 1679309956754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:16,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:16,758 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:16,758 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:16,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:16,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 5 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52180 deadline: 1679309956757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:16,758 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:16,759 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:16,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:16,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:16,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:16,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 5 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52190 deadline: 1679309956757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:16,759 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:16,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:16,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:16,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52136 deadline: 1679309956758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:16,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:16,759 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:16,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:16,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:16,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:16,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:16,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:16,761 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:16,761 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:16,761 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:16,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:16,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:16,761 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:16,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:16,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:16,761 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:16,761 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:16,762 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:16,762 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:16,762 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:16,762 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:16,762 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:16,762 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:16,762 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:16,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52164 deadline: 1679309956762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:16,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:16,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:16,765 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:16,862 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:16,862 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:16,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52176 deadline: 1679309956859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:16,862 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:16,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52180 deadline: 1679309956859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:16,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:16,863 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:16,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52190 deadline: 1679309956860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:16,863 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:16,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52136 deadline: 1679309956860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:16,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:16,864 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:16,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52164 deadline: 1679309956863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:16,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:16,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:16,870 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:16,870 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:16,870 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:16,870 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:16,871 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:16,871 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:16,871 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:16,871 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:16,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:16,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:16,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:16,872 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:16,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:16,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:16,873 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:16,873 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:16,873 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:16,873 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:16,873 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:16,873 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:16,873 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:16,873 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:16,878 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:16,878 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:16,878 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:16,878 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:16,878 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. as already flushing 2023-03-20 10:58:16,878 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:16,878 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:16,878 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:16,878 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:16,886 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:16,887 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:16,889 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:16,889 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:16,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:16,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:16,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:16,889 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:16,889 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:16,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:16,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:16,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:16,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:16,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:16,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:16,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2066678a[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:16,893 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:16,893 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:16,893 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:16,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:16,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:16,894 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:16,894 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:16,894 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:16,894 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:16,894 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:16,894 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:16,895 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:16,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:16,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:16,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:16,895 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:16,895 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:16,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:16,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:16,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:16,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:16,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:16,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:16,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:16,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:16,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:16,896 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:16,896 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:16,896 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:16,896 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:16,896 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:16,897 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:16,897 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:16,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:16,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:16,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:16,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:16,897 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:16,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:16,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:16,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:16,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:16,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:16,898 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:16,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:16,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:16,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:16,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:16,898 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:16,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:16,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:16,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:16,902 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:16,902 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:16,902 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:16,902 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:16,902 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:16,902 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:16,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:16,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:16,902 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:16,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:16,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:16,902 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:16,902 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:16,902 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:16,902 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:16,903 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:16,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:16,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:17,003 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:17,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:17,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:17,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:17,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:17,010 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:17,010 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:17,010 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:17,010 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:17,011 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:17,011 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:17,011 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:17,011 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:17,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:17,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:17,012 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:17,012 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,012 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:17,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:17,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:17,012 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:17,013 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:17,013 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:17,013 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:17,013 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:17,013 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:17,013 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:17,013 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:17,013 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:17,013 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:17,013 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. as already flushing 2023-03-20 10:58:17,013 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:17,013 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:17,013 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:17,014 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,014 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:17,014 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:17,015 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,015 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:17,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:17,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:17,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:17,015 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:17,015 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:17,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:17,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:17,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:17,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:17,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@494924ee[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:17,017 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:17,017 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:17,017 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:17,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:17,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:17,018 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:17,018 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:17,018 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:17,018 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,018 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:17,019 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,019 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:17,019 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:17,019 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:17,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:17,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:17,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:17,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:17,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:17,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:17,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:17,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:17,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:17,022 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:17,022 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:17,022 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:17,023 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:17,023 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:17,024 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:17,024 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:17,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:17,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:17,024 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:17,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:17,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:17,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:17,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:17,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:17,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:17,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:17,025 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:17,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:17,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:17,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,026 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:17,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:17,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,028 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:17,028 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:17,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:17,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:17,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:17,028 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:17,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:17,028 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,028 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:17,029 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:17,029 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:17,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:17,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:17,029 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:17,029 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:17,029 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,029 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:17,029 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:17,029 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:17,066 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:17,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52176 deadline: 1679309957063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,066 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:17,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52190 deadline: 1679309957064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,066 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:17,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52136 deadline: 1679309957064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,066 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:17,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52180 deadline: 1679309957064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,079 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:17,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52164 deadline: 1679309957065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,129 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:17,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:17,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:17,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:17,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:17,132 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:17,132 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:17,132 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:17,132 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:17,132 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:17,132 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:17,133 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,133 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:17,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:17,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:17,133 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,133 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:17,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:17,134 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:17,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:17,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:17,134 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:17,134 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:17,134 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:17,134 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:17,134 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:17,137 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:17,137 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:17,137 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:17,137 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:17,137 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. as already flushing 2023-03-20 10:58:17,137 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:17,137 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:17,137 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:17,137 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,137 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:17,137 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:17,144 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:17,144 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:17,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:17,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:17,146 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:17,146 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:17,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:17,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:17,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:17,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:17,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a53277[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:17,148 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:17,148 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:17,153 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:17,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:17,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:17,153 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:17,153 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,154 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:17,153 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:17,154 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:17,161 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:17,161 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:17,161 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:17,162 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:17,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:17,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:17,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:17,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:17,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:17,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:17,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:17,164 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:17,164 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:17,164 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:17,164 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:17,164 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:17,164 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:17,168 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/A/e7708c54b7694de68311caada8bc0732 2023-03-20 10:58:17,169 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:17,169 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:17,169 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:17,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:17,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:17,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:17,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:17,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:17,169 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:17,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:17,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:17,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:17,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:17,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:17,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:17,171 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:17,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:17,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,178 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,178 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:17,178 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:17,178 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,178 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:17,178 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:17,178 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:17,178 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:17,178 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:17,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:17,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:17,178 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:17,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:17,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:17,178 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:17,178 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:17,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:17,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:17,179 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:17,197 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/B/e73d1f22ec8d48ef99ee60c3ad764200 2023-03-20 10:58:17,252 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/C/4cd7075d7aff49879958571076203591 2023-03-20 10:58:17,257 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/A/e7708c54b7694de68311caada8bc0732 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/e7708c54b7694de68311caada8bc0732 2023-03-20 10:58:17,263 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/e7708c54b7694de68311caada8bc0732, entries=100, sequenceid=13, filesize=9.3 K 2023-03-20 10:58:17,263 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/B/e73d1f22ec8d48ef99ee60c3ad764200 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/e73d1f22ec8d48ef99ee60c3ad764200 2023-03-20 10:58:17,270 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/e73d1f22ec8d48ef99ee60c3ad764200, entries=100, sequenceid=13, filesize=9.3 K 2023-03-20 10:58:17,271 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/C/4cd7075d7aff49879958571076203591 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/4cd7075d7aff49879958571076203591 2023-03-20 10:58:17,276 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/4cd7075d7aff49879958571076203591, entries=100, sequenceid=13, filesize=9.3 K 2023-03-20 10:58:17,277 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 00ef1cabe628ad083ccaa287ccbbfb01 in 538ms, sequenceid=13, compaction requested=false 2023-03-20 10:58:17,277 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 00ef1cabe628ad083ccaa287ccbbfb01: 2023-03-20 10:58:17,278 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:17,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:17,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:17,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:17,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:17,288 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:17,288 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:17,288 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:17,288 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:17,289 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:17,289 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:17,289 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,289 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:17,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:17,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:17,289 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,290 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:17,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:17,290 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:17,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:17,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:17,290 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:17,290 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:17,290 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:17,290 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:17,290 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:17,290 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:17,290 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:17,291 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:17,291 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:17,291 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:17,291 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 00ef1cabe628ad083ccaa287ccbbfb01 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-20 10:58:17,292 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=A 2023-03-20 10:58:17,292 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:17,292 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=B 2023-03-20 10:58:17,292 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:17,292 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=C 2023-03-20 10:58:17,292 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:17,312 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/A/b7644e97e9f44f5d816e140bbe3dafca 2023-03-20 10:58:17,322 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/B/f92efa50737c44c3842a479aefb28315 2023-03-20 10:58:17,371 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. as already flushing 2023-03-20 10:58:17,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 00ef1cabe628ad083ccaa287ccbbfb01 2023-03-20 10:58:17,391 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:17,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52190 deadline: 1679309957383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,391 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:17,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52180 deadline: 1679309957386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,395 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:17,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52176 deadline: 1679309957390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,396 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:17,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52164 deadline: 1679309957391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,396 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:17,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52136 deadline: 1679309957391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,496 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:17,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52180 deadline: 1679309957492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,497 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:17,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52190 deadline: 1679309957492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,501 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:17,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52176 deadline: 1679309957497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,501 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:17,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52136 deadline: 1679309957497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,501 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:17,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52164 deadline: 1679309957498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,699 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:17,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52180 deadline: 1679309957698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,708 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:17,708 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:17,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52176 deadline: 1679309957702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52136 deadline: 1679309957703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,708 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:17,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52164 deadline: 1679309957703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,712 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:17,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52190 deadline: 1679309957706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,781 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/C/c4522be77b084496852c128262f434d9 2023-03-20 10:58:17,785 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/A/b7644e97e9f44f5d816e140bbe3dafca as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/b7644e97e9f44f5d816e140bbe3dafca 2023-03-20 10:58:17,788 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/b7644e97e9f44f5d816e140bbe3dafca, entries=150, sequenceid=33, filesize=11.6 K 2023-03-20 10:58:17,789 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/B/f92efa50737c44c3842a479aefb28315 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/f92efa50737c44c3842a479aefb28315 2023-03-20 10:58:17,792 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/f92efa50737c44c3842a479aefb28315, entries=150, sequenceid=33, filesize=11.6 K 2023-03-20 10:58:17,795 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/C/c4522be77b084496852c128262f434d9 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/c4522be77b084496852c128262f434d9 2023-03-20 10:58:17,800 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/c4522be77b084496852c128262f434d9, entries=150, sequenceid=33, filesize=11.6 K 2023-03-20 10:58:17,801 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for 00ef1cabe628ad083ccaa287ccbbfb01 in 510ms, sequenceid=33, compaction requested=false 2023-03-20 10:58:17,801 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 00ef1cabe628ad083ccaa287ccbbfb01: 2023-03-20 10:58:17,801 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:17,801 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:17,801 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:17,801 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,801 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:17,801 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:17,802 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,802 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:17,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:17,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:17,803 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:17,803 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:17,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:17,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:17,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:17,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:17,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:17,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3be85aa9[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:17,822 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:17,822 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:17,824 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:17,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:17,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:17,824 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:17,825 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:17,825 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:17,825 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,825 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:17,825 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:17,825 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:17,825 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:17,826 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:17,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:17,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:17,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:17,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:17,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:17,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:17,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:17,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:17,830 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:17,830 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:17,830 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:17,830 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:17,830 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:17,836 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:17,836 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:17,836 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:17,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:17,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:17,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:17,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:17,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:17,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:17,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:17,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:17,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:17,840 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:17,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:17,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:17,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,841 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:17,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:17,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,845 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:17,845 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:17,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:17,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:17,846 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,846 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:17,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:17,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:17,846 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:17,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:17,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:17,846 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:17,846 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,846 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:17,846 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:17,846 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:17,846 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:17,846 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:17,946 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:17,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:17,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:17,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:17,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:17,949 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:17,949 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:17,950 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:17,950 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:17,950 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:17,950 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:17,951 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:17,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:17,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:17,951 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:17,952 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:17,952 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:17,952 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:17,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:17,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:17,952 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:17,952 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:17,952 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:17,952 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:17,952 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:17,952 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:17,954 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:17,954 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:17,954 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:17,954 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:17,954 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 00ef1cabe628ad083ccaa287ccbbfb01 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-20 10:58:17,954 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=A 2023-03-20 10:58:17,954 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:17,954 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=B 2023-03-20 10:58:17,954 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:17,955 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=C 2023-03-20 10:58:17,955 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:18,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 00ef1cabe628ad083ccaa287ccbbfb01 2023-03-20 10:58:18,005 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. as already flushing 2023-03-20 10:58:18,031 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:18,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52176 deadline: 1679309958023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:18,035 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:18,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52180 deadline: 1679309958027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:18,036 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:18,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52136 deadline: 1679309958028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:18,041 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:18,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52190 deadline: 1679309958031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:18,041 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:18,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52164 deadline: 1679309958031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:18,141 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:18,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52176 deadline: 1679309958135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:18,147 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:18,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52180 deadline: 1679309958142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:18,148 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:18,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52164 deadline: 1679309958142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:18,148 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:18,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52190 deadline: 1679309958143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:18,153 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:18,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52136 deadline: 1679309958146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:18,346 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:18,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52176 deadline: 1679309958344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:18,355 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:18,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52164 deadline: 1679309958349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:18,356 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:18,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52190 deadline: 1679309958349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:18,356 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:18,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52180 deadline: 1679309958349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:18,365 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:18,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52136 deadline: 1679309958357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:18,373 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/A/109b7c544e0f4d6baf81fd6b5d6b8a9c 2023-03-20 10:58:18,390 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/B/715a01a8cd4f4b45bcb8fb5e74535872 2023-03-20 10:58:18,422 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/C/ca97ba0489744cfe8516c598d74ddddc 2023-03-20 10:58:18,429 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/A/109b7c544e0f4d6baf81fd6b5d6b8a9c as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/109b7c544e0f4d6baf81fd6b5d6b8a9c 2023-03-20 10:58:18,433 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/109b7c544e0f4d6baf81fd6b5d6b8a9c, entries=150, sequenceid=46, filesize=11.6 K 2023-03-20 10:58:18,437 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/B/715a01a8cd4f4b45bcb8fb5e74535872 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/715a01a8cd4f4b45bcb8fb5e74535872 2023-03-20 10:58:18,448 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/715a01a8cd4f4b45bcb8fb5e74535872, entries=150, sequenceid=46, filesize=11.6 K 2023-03-20 10:58:18,448 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/C/ca97ba0489744cfe8516c598d74ddddc as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/ca97ba0489744cfe8516c598d74ddddc 2023-03-20 10:58:18,456 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/ca97ba0489744cfe8516c598d74ddddc, entries=150, sequenceid=46, filesize=11.6 K 2023-03-20 10:58:18,456 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 00ef1cabe628ad083ccaa287ccbbfb01 in 502ms, sequenceid=46, compaction requested=true 2023-03-20 10:58:18,457 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 00ef1cabe628ad083ccaa287ccbbfb01: 2023-03-20 10:58:18,457 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:18,457 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:18,457 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:18,457 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:18,457 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:18,457 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:18,458 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:18,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:18,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:18,458 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:18,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:18,458 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:18,459 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:18,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:18,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:18,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:18,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:18,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:18,462 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:18,462 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28212069[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:18,462 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:18,462 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:18,467 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:18,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:18,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:18,467 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:18,467 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:18,467 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:18,468 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:18,468 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:18,468 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:18,469 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:18,469 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:18,469 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:18,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:18,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:18,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:18,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:18,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:18,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:18,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:18,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:18,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:18,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:18,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:18,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:18,473 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:18,473 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:18,473 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:18,473 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:18,473 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:18,474 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:18,474 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:18,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:18,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:18,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:18,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:18,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:18,476 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:18,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:18,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:18,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:18,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:18,477 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:18,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:18,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:18,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:18,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:18,480 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:18,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:18,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:18,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:18,482 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:18,482 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:18,482 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:18,482 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:18,482 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:18,482 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:18,483 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:18,482 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:18,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:18,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:18,483 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:18,483 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:18,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:18,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:18,484 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:18,484 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:18,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:18,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:18,584 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:18,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:18,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:18,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:18,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:18,602 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:18,602 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:18,602 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:18,602 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:18,603 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:18,603 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:18,604 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:18,604 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:18,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:18,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:18,604 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:18,604 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:18,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:18,606 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:18,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:18,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:18,606 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:18,614 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:18,614 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:18,614 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:18,614 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:18,614 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:18,614 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:18,614 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:18,614 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:18,614 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:18,615 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 00ef1cabe628ad083ccaa287ccbbfb01 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-20 10:58:18,615 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=A 2023-03-20 10:58:18,615 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:18,615 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=B 2023-03-20 10:58:18,615 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:18,615 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=C 2023-03-20 10:58:18,615 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:18,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 00ef1cabe628ad083ccaa287ccbbfb01 2023-03-20 10:58:18,653 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. as already flushing 2023-03-20 10:58:18,670 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:18,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52180 deadline: 1679309958664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:18,670 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:18,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52136 deadline: 1679309958667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:18,674 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:18,674 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:18,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52164 deadline: 1679309958668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:18,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52190 deadline: 1679309958669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:18,674 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:18,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52176 deadline: 1679309958670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:18,774 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:18,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52180 deadline: 1679309958771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:18,775 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:18,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52176 deadline: 1679309958775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:18,775 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:18,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52164 deadline: 1679309958775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:18,779 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:18,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52190 deadline: 1679309958776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:18,979 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:18,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52180 deadline: 1679309958976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:18,984 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:18,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52176 deadline: 1679309958977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:18,984 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:18,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52164 deadline: 1679309958977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:18,985 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:18,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52190 deadline: 1679309958981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:19,029 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=65 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/A/f901a2b38200433789bca7579863abef 2023-03-20 10:58:19,128 DEBUG [regionserver/jenkins-hbase20:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-03-20 10:58:19,179 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:19,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52136 deadline: 1679309959173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:19,285 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:19,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52180 deadline: 1679309959281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:19,292 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:19,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52190 deadline: 1679309959287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:19,292 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:19,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52176 deadline: 1679309959288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:19,293 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:19,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52164 deadline: 1679309959288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:19,438 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=65 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/B/788d0d925cb0412d8c2ec20e8ca2b149 2023-03-20 10:58:19,476 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=65 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/C/c63dc7ef0cf14aeb999cbc9bf5f53846 2023-03-20 10:58:19,492 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/A/f901a2b38200433789bca7579863abef as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/f901a2b38200433789bca7579863abef 2023-03-20 10:58:19,506 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/f901a2b38200433789bca7579863abef, entries=150, sequenceid=65, filesize=11.6 K 2023-03-20 10:58:19,509 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/B/788d0d925cb0412d8c2ec20e8ca2b149 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/788d0d925cb0412d8c2ec20e8ca2b149 2023-03-20 10:58:19,518 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/788d0d925cb0412d8c2ec20e8ca2b149, entries=150, sequenceid=65, filesize=11.6 K 2023-03-20 10:58:19,523 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/C/c63dc7ef0cf14aeb999cbc9bf5f53846 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/c63dc7ef0cf14aeb999cbc9bf5f53846 2023-03-20 10:58:19,530 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/c63dc7ef0cf14aeb999cbc9bf5f53846, entries=150, sequenceid=65, filesize=11.6 K 2023-03-20 10:58:19,533 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 00ef1cabe628ad083ccaa287ccbbfb01 in 918ms, sequenceid=65, compaction requested=true 2023-03-20 10:58:19,533 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 00ef1cabe628ad083ccaa287ccbbfb01: 2023-03-20 10:58:19,533 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:19,533 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:19,533 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:19,534 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:19,546 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:19,546 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:19,567 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:19,567 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:19,567 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:19,570 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:19,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:19,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:19,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:19,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:19,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:19,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:19,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:19,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:19,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:19,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28cf1fd7[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:19,576 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:19,576 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:19,580 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:19,580 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:19,582 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:19,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:19,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:19,582 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:19,582 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:19,582 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:19,586 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:19,586 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:19,586 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:19,586 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:19,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:19,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:19,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:19,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:19,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:19,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:19,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:19,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:19,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:19,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:19,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:19,591 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:19,591 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:19,591 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:19,591 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:19,591 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:19,591 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:19,594 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:19,594 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:19,594 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:19,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:19,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:19,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:19,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:19,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:19,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:19,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:19,595 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:19,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:19,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:19,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:19,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:19,604 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:19,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:19,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:19,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:19,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:19,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:19,607 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:19,607 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:19,607 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:19,607 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:19,607 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:19,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:19,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:19,607 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:19,607 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:19,607 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:19,607 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:19,607 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:19,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:19,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:19,608 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:19,608 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:19,609 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:19,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:19,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:19,708 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:19,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:19,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:19,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:19,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:19,711 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:19,711 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:19,711 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:19,711 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:19,711 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:19,711 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:19,712 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:19,712 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:19,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:19,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:19,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:19,712 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:19,713 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:19,713 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:19,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:19,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:19,713 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:19,713 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:19,713 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:19,713 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:19,713 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:19,713 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:19,714 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:19,714 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:19,714 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:19,714 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:19,714 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 00ef1cabe628ad083ccaa287ccbbfb01 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-20 10:58:19,714 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=A 2023-03-20 10:58:19,714 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:19,714 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=B 2023-03-20 10:58:19,715 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:19,715 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=C 2023-03-20 10:58:19,715 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:19,790 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-20 10:58:19,799 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. as already flushing 2023-03-20 10:58:19,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 00ef1cabe628ad083ccaa287ccbbfb01 2023-03-20 10:58:19,866 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:19,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52180 deadline: 1679309959851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:19,866 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:19,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52176 deadline: 1679309959853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:19,873 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:19,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52164 deadline: 1679309959868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:19,873 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:19,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52190 deadline: 1679309959868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:19,974 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:19,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52176 deadline: 1679309959968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:19,979 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:19,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52180 deadline: 1679309959969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:19,988 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:19,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52190 deadline: 1679309959975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:19,989 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:19,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52164 deadline: 1679309959975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:20,157 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=78 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/A/236e21770d6445d6afdf1065d7372524 2023-03-20 10:58:20,186 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:20,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52176 deadline: 1679309960176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:20,191 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:20,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52180 deadline: 1679309960181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:20,193 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:20,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52136 deadline: 1679309960184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:20,198 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:20,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52190 deadline: 1679309960190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:20,199 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:20,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52164 deadline: 1679309960192, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:20,492 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:20,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52176 deadline: 1679309960488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:20,499 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:20,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52180 deadline: 1679309960492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:20,504 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:20,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52164 deadline: 1679309960503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:20,504 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:20,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52190 deadline: 1679309960503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:20,634 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=78 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/B/fb5e64fd6a244eaba8f99f120d121c1b 2023-03-20 10:58:21,004 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:21,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52176 deadline: 1679309960996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,009 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:21,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52180 deadline: 1679309961001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,013 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:21,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52190 deadline: 1679309961008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,013 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:21,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52164 deadline: 1679309961008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,078 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=78 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/C/04b8974f07564d7a8c6c9150497ab5a4 2023-03-20 10:58:21,084 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/A/236e21770d6445d6afdf1065d7372524 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/236e21770d6445d6afdf1065d7372524 2023-03-20 10:58:21,088 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/236e21770d6445d6afdf1065d7372524, entries=150, sequenceid=78, filesize=11.6 K 2023-03-20 10:58:21,088 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/B/fb5e64fd6a244eaba8f99f120d121c1b as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/fb5e64fd6a244eaba8f99f120d121c1b 2023-03-20 10:58:21,123 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/fb5e64fd6a244eaba8f99f120d121c1b, entries=150, sequenceid=78, filesize=11.6 K 2023-03-20 10:58:21,127 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/C/04b8974f07564d7a8c6c9150497ab5a4 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/04b8974f07564d7a8c6c9150497ab5a4 2023-03-20 10:58:21,142 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/04b8974f07564d7a8c6c9150497ab5a4, entries=150, sequenceid=78, filesize=11.6 K 2023-03-20 10:58:21,143 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 00ef1cabe628ad083ccaa287ccbbfb01 in 1429ms, sequenceid=78, compaction requested=true 2023-03-20 10:58:21,143 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 00ef1cabe628ad083ccaa287ccbbfb01: 2023-03-20 10:58:21,143 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:21,143 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:21,143 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:21,143 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,143 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:21,143 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:21,145 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:21,145 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:21,145 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:21,150 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:21,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:21,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:21,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:21,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:21,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:21,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:21,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@455b633[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:21,152 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:21,152 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:21,160 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:21,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:21,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:21,161 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:21,161 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,161 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:21,161 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:21,161 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:21,162 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:21,162 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:21,162 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:21,162 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:21,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:21,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:21,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:21,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:21,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:21,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:21,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:21,167 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:21,167 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:21,167 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:21,167 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:21,167 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:21,167 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:21,178 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:21,178 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:21,179 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:21,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:21,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:21,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:21,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:21,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:21,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:21,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:21,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:21,179 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:21,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:21,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:21,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:21,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,180 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:21,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:21,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,182 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,182 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:21,182 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:21,182 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:21,182 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:21,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:21,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:21,182 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,182 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:21,182 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:21,182 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:21,182 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:21,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:21,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:21,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:21,183 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:21,183 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:21,183 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:21,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:21,284 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:21,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:21,286 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:21,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:21,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:21,289 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:21,290 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:21,290 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:21,290 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:21,290 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:21,290 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:21,293 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:21,293 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:21,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:21,294 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,294 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:21,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:21,294 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:21,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:21,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:21,295 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:21,296 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:21,296 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:21,296 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:21,296 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:21,296 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:21,296 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:21,296 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:21,296 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:21,296 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:21,296 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 00ef1cabe628ad083ccaa287ccbbfb01 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-20 10:58:21,296 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=A 2023-03-20 10:58:21,297 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:21,297 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=B 2023-03-20 10:58:21,297 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:21,297 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=C 2023-03-20 10:58:21,297 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:21,307 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=97 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/A/b6592faf46c94b4e92aa4f9ae553140f 2023-03-20 10:58:21,348 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=97 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/B/4baa322043ce48a6979a83563875bf6d 2023-03-20 10:58:21,398 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=97 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/C/bc20d1eeffd3454388980049639db488 2023-03-20 10:58:21,409 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/A/b6592faf46c94b4e92aa4f9ae553140f as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/b6592faf46c94b4e92aa4f9ae553140f 2023-03-20 10:58:21,412 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/b6592faf46c94b4e92aa4f9ae553140f, entries=150, sequenceid=97, filesize=11.6 K 2023-03-20 10:58:21,413 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/B/4baa322043ce48a6979a83563875bf6d as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/4baa322043ce48a6979a83563875bf6d 2023-03-20 10:58:21,423 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/4baa322043ce48a6979a83563875bf6d, entries=150, sequenceid=97, filesize=11.6 K 2023-03-20 10:58:21,424 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/C/bc20d1eeffd3454388980049639db488 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/bc20d1eeffd3454388980049639db488 2023-03-20 10:58:21,428 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/bc20d1eeffd3454388980049639db488, entries=150, sequenceid=97, filesize=11.6 K 2023-03-20 10:58:21,429 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=0 B/0 for 00ef1cabe628ad083ccaa287ccbbfb01 in 133ms, sequenceid=97, compaction requested=true 2023-03-20 10:58:21,429 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 00ef1cabe628ad083ccaa287ccbbfb01: 2023-03-20 10:58:21,429 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:21,429 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:21,429 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:21,429 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,429 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:21,429 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:21,431 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:21,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:21,431 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:21,432 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:21,432 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:21,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:21,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:21,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:21,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:21,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:21,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d919902[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:21,435 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:21,435 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:21,435 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:21,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:21,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:21,436 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:21,436 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,436 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:21,436 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:21,436 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:21,437 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,437 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:21,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:21,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:21,437 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:21,437 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:21,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:21,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:21,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:21,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:21,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:21,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:21,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:21,440 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:21,441 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:21,441 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:21,441 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:21,441 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:21,460 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:21,460 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:21,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:21,461 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:21,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:21,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:21,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:21,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:21,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:21,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:21,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:21,461 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:21,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:21,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:21,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:21,463 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:21,465 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:21,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:21,465 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:21,465 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:21,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:21,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:21,465 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:21,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:21,465 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:21,465 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:21,465 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,465 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:21,465 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:21,465 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:21,465 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:21,466 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:21,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:21,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:21,466 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:21,566 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:21,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:21,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:21,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:21,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:21,573 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:21,573 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:21,573 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:21,573 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:21,575 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:21,575 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:21,576 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:21,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:21,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:21,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:21,577 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,577 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:21,577 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,577 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:21,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:21,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:21,578 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:21,583 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:21,586 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:21,586 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:21,586 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:21,586 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:21,601 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:21,601 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:21,601 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:21,601 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:21,601 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 00ef1cabe628ad083ccaa287ccbbfb01: 2023-03-20 10:58:21,601 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:21,601 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:21,601 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:21,601 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,601 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:21,601 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:21,603 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:21,603 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:21,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:21,603 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:21,603 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:21,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:21,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:21,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:21,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:21,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:21,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c92ccb1[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:21,606 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:21,606 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:21,607 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:21,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:21,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:21,608 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:21,608 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:21,608 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:21,607 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,608 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:21,609 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:21,609 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:21,609 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:21,609 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:21,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:21,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:21,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:21,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:21,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:21,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:21,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:21,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:21,614 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:21,614 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:21,614 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:21,614 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:21,614 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:21,616 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:21,616 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:21,616 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:21,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:21,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:21,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:21,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:21,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:21,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:21,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:21,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:21,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:21,619 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:21,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:21,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:21,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,624 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:21,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:21,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,627 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,628 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:21,628 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:21,628 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,628 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:21,628 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:21,628 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:21,628 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:21,628 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:21,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:21,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:21,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:21,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:21,628 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:21,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:21,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:21,629 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:21,629 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:21,729 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:21,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:21,734 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:21,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:21,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:21,739 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:21,739 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:21,739 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:21,739 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:21,740 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:21,741 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:21,741 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,741 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:21,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:21,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:21,742 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,742 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:21,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:21,742 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:21,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:21,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:21,742 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:21,742 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:21,742 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:21,742 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:21,742 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:21,742 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:21,746 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:21,746 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:21,746 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:21,746 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:21,746 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 00ef1cabe628ad083ccaa287ccbbfb01: 2023-03-20 10:58:21,746 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:21,746 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:21,746 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:21,746 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,746 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:21,746 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:21,747 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:21,747 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:21,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:21,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:21,747 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:21,747 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:21,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:21,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:21,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:21,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:21,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c734153[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:21,748 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:21,748 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:21,749 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:21,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:21,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:21,749 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:21,749 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,749 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:21,749 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:21,749 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:21,750 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:21,750 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:21,750 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:21,751 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:21,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:21,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:21,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:21,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:21,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:21,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:21,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:21,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:21,752 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:21,752 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:21,752 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:21,752 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:21,752 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:21,753 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:21,753 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:21,753 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:21,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:21,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:21,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:21,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:21,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:21,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:21,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:21,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:21,754 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:21,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:21,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:21,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:21,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,756 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:21,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:21,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,770 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,770 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:21,770 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:21,770 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:21,770 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:21,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:21,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:21,770 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:21,770 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,770 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:21,770 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:21,770 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:21,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:21,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:21,770 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:21,770 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:21,770 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:21,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:21,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:21,871 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:21,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:21,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:21,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:21,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:21,873 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:21,873 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:21,873 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:21,873 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:21,873 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:21,873 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:21,874 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,874 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:21,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:21,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:21,874 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,874 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:21,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:21,875 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:21,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:21,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:21,875 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:21,876 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:21,876 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:21,876 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:21,876 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:21,876 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:21,876 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:21,876 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:21,876 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:21,876 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:21,877 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 00ef1cabe628ad083ccaa287ccbbfb01: 2023-03-20 10:58:21,877 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:21,877 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:21,877 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:21,877 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,877 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:21,877 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:21,878 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,878 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:21,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:21,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:21,879 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:21,879 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:21,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:21,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:21,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:21,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:21,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:21,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@253e8ac0[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:21,880 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:21,880 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:21,880 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:21,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:21,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:21,881 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:21,881 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:21,881 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:21,881 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,881 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:21,881 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,881 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:21,881 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:21,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:21,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:21,881 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:21,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:21,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:21,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:21,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:21,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:21,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:21,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:21,884 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:21,884 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:21,884 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:21,884 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:21,884 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:21,885 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:21,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:21,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:21,885 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:21,885 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:21,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:21,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:21,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:21,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:21,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:21,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:21,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:21,890 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:21,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:21,891 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:21,892 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:21,895 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:21,895 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:21,895 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:21,895 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:21,895 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:21,895 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:21,895 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:21,896 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:21,895 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:21,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:21,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:21,897 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:21,897 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:21,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:21,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:21,896 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:21,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:21,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:21,997 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:22,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:22,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:22,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:22,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:22,026 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:22,026 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:22,027 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:22,027 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:22,028 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 00ef1cabe628ad083ccaa287ccbbfb01 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-20 10:58:22,029 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=A 2023-03-20 10:58:22,029 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:22,029 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=B 2023-03-20 10:58:22,029 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:22,029 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=C 2023-03-20 10:58:22,029 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:22,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 00ef1cabe628ad083ccaa287ccbbfb01 2023-03-20 10:58:22,030 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:22,030 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:22,031 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:22,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:22,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:22,031 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:22,032 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:22,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:22,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:22,032 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:22,032 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,033 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:22,034 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:22,034 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:22,034 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:22,034 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:22,034 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:22,034 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:22,035 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:22,035 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:22,035 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:22,035 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. as already flushing 2023-03-20 10:58:22,035 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:22,035 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:22,035 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:22,035 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,035 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:22,035 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:22,044 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=110 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/A/e69abef3b8024b1a9ae3b3ae61d4ac46 2023-03-20 10:58:22,044 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,044 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:22,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:22,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:22,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:22,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:22,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:22,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:22,046 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:22,046 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:22,046 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54164eec[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:22,046 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:22,047 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,047 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:22,048 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:22,048 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:22,048 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:22,048 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:22,048 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:22,048 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:22,061 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:22,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:22,063 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:22,063 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:22,063 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:22,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:22,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:22,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:22,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:22,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:22,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:22,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:22,067 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:22,067 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:22,067 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:22,067 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:22,067 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:22,070 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:22,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:22,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:22,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:22,070 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:22,070 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:22,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:22,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:22,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:22,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:22,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:22,071 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:22,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:22,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:22,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:22,072 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,072 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=110 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/B/bbf408bb9414416d922cb9b4cfcca676 2023-03-20 10:58:22,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:22,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:22,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,084 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,084 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:22,084 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:22,085 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,085 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:22,085 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:22,084 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:22,085 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:22,085 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:22,085 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:22,085 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:22,085 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:22,085 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:22,086 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:22,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:22,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:22,085 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:22,088 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:22,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:22,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:22,095 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1679309902062, max timestamp=1679309902092Num uniques 859; 2023-03-20 10:58:22,095 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-03-20 10:58:22,095 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1679309902062, max timestamp=1679309902092Num uniques 859; 2023-03-20 10:58:22,095 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-03-20 10:58:22,096 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1679309902062, max timestamp=1679309902092Num uniques 859; 2023-03-20 10:58:22,096 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-03-20 10:58:22,134 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:22,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52176 deadline: 1679309962128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,137 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:22,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52180 deadline: 1679309962129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,139 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:22,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52190 deadline: 1679309962134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,142 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:22,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52164 deadline: 1679309962135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,186 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:22,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:22,188 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:22,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:22,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:22,190 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:22,190 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:22,190 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:22,190 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:22,191 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:22,191 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:22,191 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,191 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:22,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:22,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:22,192 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,192 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:22,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:22,192 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:22,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:22,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:22,192 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:22,193 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:22,193 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:22,193 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:22,193 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:22,193 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:22,194 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:22,194 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:22,194 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:22,194 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:22,194 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. as already flushing 2023-03-20 10:58:22,194 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:22,194 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:22,195 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:22,195 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,195 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:22,195 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:22,207 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:22,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52136 deadline: 1679309962200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,209 DEBUG [Thread-2963] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4180 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01., hostname=jenkins-hbase20.apache.org,36041,1679309777062, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-20 10:58:22,211 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:22,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:22,211 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:22,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:22,211 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:22,211 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:22,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:22,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:22,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:22,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:22,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1eabe79b[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:22,215 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:22,215 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:22,219 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:22,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:22,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:22,219 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:22,219 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:22,219 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,220 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:22,219 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:22,221 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:22,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:22,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:22,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:22,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:22,222 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:22,222 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:22,222 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:22,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:22,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:22,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:22,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:22,225 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:22,225 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:22,225 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:22,225 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:22,225 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:22,226 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:22,226 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:22,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:22,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:22,230 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:22,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:22,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:22,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:22,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:22,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:22,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:22,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:22,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:22,236 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:22,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:22,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,237 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:22,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:22,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,244 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:22,244 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:22,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:22,245 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:22,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:22,245 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:22,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:22,245 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:22,245 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:22,245 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,245 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:22,245 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:22,245 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:22,245 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:22,245 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:22,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:22,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:22,248 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:22,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52176 deadline: 1679309962236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,249 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:22,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52180 deadline: 1679309962238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,249 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:22,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52190 deadline: 1679309962240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,249 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:22,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52164 deadline: 1679309962243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,345 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:22,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:22,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:22,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:22,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:22,348 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:22,348 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:22,348 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:22,348 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:22,349 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:22,349 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:22,349 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,349 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:22,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:22,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:22,349 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,349 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:22,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:22,350 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:22,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:22,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:22,350 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:22,350 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:22,350 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:22,350 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:22,350 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:22,350 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:22,351 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:22,351 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:22,351 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:22,351 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:22,351 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. as already flushing 2023-03-20 10:58:22,351 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:22,351 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:22,351 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:22,351 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,351 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:22,351 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:22,352 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:22,352 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:22,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:22,352 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:22,352 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:22,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:22,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:22,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:22,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:22,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:22,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@84454c4[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:22,353 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:22,353 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:22,354 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:22,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:22,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:22,354 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:22,354 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:22,354 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:22,354 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,355 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:22,356 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:22,356 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,356 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:22,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:22,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:22,356 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:22,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:22,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:22,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:22,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:22,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:22,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:22,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:22,358 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:22,358 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:22,358 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:22,358 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:22,358 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:22,359 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:22,359 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:22,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:22,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:22,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:22,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:22,359 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:22,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:22,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:22,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:22,360 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:22,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:22,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:22,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:22,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:22,360 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:22,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:22,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,362 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:22,362 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:22,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:22,362 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,362 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:22,362 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:22,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:22,362 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:22,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:22,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:22,362 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:22,362 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,362 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:22,362 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:22,362 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:22,362 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:22,362 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:22,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:22,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:22,462 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:22,463 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:22,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52176 deadline: 1679309962450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,463 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:22,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52180 deadline: 1679309962450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,463 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:22,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52164 deadline: 1679309962451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:22,463 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:22,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52190 deadline: 1679309962451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:22,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:22,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:22,466 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:22,467 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:22,467 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:22,467 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:22,467 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:22,467 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:22,469 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:22,469 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:22,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:22,470 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,470 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:22,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:22,470 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:22,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:22,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:22,470 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:22,474 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:22,474 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:22,474 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:22,474 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:22,474 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:22,482 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:22,482 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:22,482 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:22,482 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:22,482 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. as already flushing 2023-03-20 10:58:22,482 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:22,482 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:22,482 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:22,482 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,482 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:22,482 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:22,483 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,483 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:22,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:22,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:22,484 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:22,484 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:22,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:22,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:22,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:22,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:22,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:22,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@597db786[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:22,485 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:22,485 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:22,486 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:22,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:22,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:22,486 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:22,486 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,486 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:22,486 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:22,486 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:22,488 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:22,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:22,488 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:22,488 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:22,488 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:22,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:22,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:22,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:22,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:22,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:22,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:22,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:22,490 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:22,490 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:22,490 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:22,490 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:22,490 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:22,491 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:22,491 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:22,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:22,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:22,491 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:22,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:22,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:22,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:22,492 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:22,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:22,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:22,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:22,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:22,493 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:22,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:22,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:22,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:22,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,496 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:22,497 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:22,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:22,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:22,497 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,497 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:22,497 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:22,497 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,497 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:22,497 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:22,497 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:22,497 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:22,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:22,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:22,497 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:22,497 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:22,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:22,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:22,498 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:22,501 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=110 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/C/916e137f601944b5a5930ad568f2aecb 2023-03-20 10:58:22,505 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/A/e69abef3b8024b1a9ae3b3ae61d4ac46 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/e69abef3b8024b1a9ae3b3ae61d4ac46 2023-03-20 10:58:22,510 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/e69abef3b8024b1a9ae3b3ae61d4ac46, entries=250, sequenceid=110, filesize=16.2 K 2023-03-20 10:58:22,511 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/B/bbf408bb9414416d922cb9b4cfcca676 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/bbf408bb9414416d922cb9b4cfcca676 2023-03-20 10:58:22,515 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/bbf408bb9414416d922cb9b4cfcca676, entries=100, sequenceid=110, filesize=9.3 K 2023-03-20 10:58:22,516 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/C/916e137f601944b5a5930ad568f2aecb as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/916e137f601944b5a5930ad568f2aecb 2023-03-20 10:58:22,526 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/916e137f601944b5a5930ad568f2aecb, entries=100, sequenceid=110, filesize=9.3 K 2023-03-20 10:58:22,527 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 00ef1cabe628ad083ccaa287ccbbfb01 in 499ms, sequenceid=110, compaction requested=true 2023-03-20 10:58:22,527 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 00ef1cabe628ad083ccaa287ccbbfb01: 2023-03-20 10:58:22,527 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:58:22,527 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-03-20 10:58:22,527 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-03-20 10:58:22,527 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:58:22,527 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:58:22,529 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 85534 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-03-20 10:58:22,529 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 78514 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-03-20 10:58:22,529 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1912): 00ef1cabe628ad083ccaa287ccbbfb01/A is initiating minor compaction (all files) 2023-03-20 10:58:22,529 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): 00ef1cabe628ad083ccaa287ccbbfb01/B is initiating minor compaction (all files) 2023-03-20 10:58:22,529 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 00ef1cabe628ad083ccaa287ccbbfb01/A in TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:22,529 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 00ef1cabe628ad083ccaa287ccbbfb01/B in TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:22,529 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/e73d1f22ec8d48ef99ee60c3ad764200, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/f92efa50737c44c3842a479aefb28315, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/715a01a8cd4f4b45bcb8fb5e74535872, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/788d0d925cb0412d8c2ec20e8ca2b149, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/fb5e64fd6a244eaba8f99f120d121c1b, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/4baa322043ce48a6979a83563875bf6d, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/bbf408bb9414416d922cb9b4cfcca676] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp, totalSize=76.7 K 2023-03-20 10:58:22,529 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/e7708c54b7694de68311caada8bc0732, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/b7644e97e9f44f5d816e140bbe3dafca, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/109b7c544e0f4d6baf81fd6b5d6b8a9c, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/f901a2b38200433789bca7579863abef, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/236e21770d6445d6afdf1065d7372524, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/b6592faf46c94b4e92aa4f9ae553140f, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/e69abef3b8024b1a9ae3b3ae61d4ac46] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp, totalSize=83.5 K 2023-03-20 10:58:22,530 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting e73d1f22ec8d48ef99ee60c3ad764200, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1679309896734 2023-03-20 10:58:22,530 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting e7708c54b7694de68311caada8bc0732, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1679309896734 2023-03-20 10:58:22,530 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting f92efa50737c44c3842a479aefb28315, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1679309896746 2023-03-20 10:58:22,530 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting b7644e97e9f44f5d816e140bbe3dafca, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1679309896746 2023-03-20 10:58:22,531 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 715a01a8cd4f4b45bcb8fb5e74535872, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1679309897377 2023-03-20 10:58:22,531 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 109b7c544e0f4d6baf81fd6b5d6b8a9c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1679309897377 2023-03-20 10:58:22,531 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 788d0d925cb0412d8c2ec20e8ca2b149, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=65, earliestPutTs=1679309898022 2023-03-20 10:58:22,531 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting f901a2b38200433789bca7579863abef, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=65, earliestPutTs=1679309898022 2023-03-20 10:58:22,531 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting fb5e64fd6a244eaba8f99f120d121c1b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=78, earliestPutTs=1679309898659 2023-03-20 10:58:22,531 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 236e21770d6445d6afdf1065d7372524, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=78, earliestPutTs=1679309898659 2023-03-20 10:58:22,532 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 4baa322043ce48a6979a83563875bf6d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1679309899849 2023-03-20 10:58:22,532 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting b6592faf46c94b4e92aa4f9ae553140f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1679309899849 2023-03-20 10:58:22,532 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting bbf408bb9414416d922cb9b4cfcca676, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=110, earliestPutTs=1679309902027 2023-03-20 10:58:22,532 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting e69abef3b8024b1a9ae3b3ae61d4ac46, keycount=250, bloomtype=ROW, size=16.2 K, encoding=NONE, compression=NONE, seqNum=110, earliestPutTs=1679309902022 2023-03-20 10:58:22,547 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): 00ef1cabe628ad083ccaa287ccbbfb01#B#compaction#404 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:58:22,551 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] throttle.PressureAwareThroughputController(145): 00ef1cabe628ad083ccaa287ccbbfb01#A#compaction#405 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:58:22,597 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:22,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:22,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:22,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:22,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:22,600 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:22,600 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:22,600 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:22,600 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:22,601 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:22,601 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:22,601 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:22,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:22,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:22,601 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:22,602 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,602 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:22,602 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:22,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:22,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:22,602 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:22,606 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:22,606 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:22,606 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:22,606 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:22,606 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:22,613 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:22,613 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:22,613 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:22,614 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 00ef1cabe628ad083ccaa287ccbbfb01 3/3 column families, dataSize=140.89 KB heapSize=386.16 KB 2023-03-20 10:58:22,614 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=A 2023-03-20 10:58:22,614 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:58:22,614 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=B 2023-03-20 10:58:22,613 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:22,614 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:58:22,615 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=C 2023-03-20 10:58:22,615 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:58:22,631 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=144 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/A/6f8722ef187e4cd8ae90dab405235d4b 2023-03-20 10:58:22,648 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=144 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/B/f82a203e14e94decb2bc0ee06b14cacb 2023-03-20 10:58:22,662 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=144 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/C/c0be06edbe84449fb269f3e4e3c2bb88 2023-03-20 10:58:22,668 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/A/6f8722ef187e4cd8ae90dab405235d4b as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/6f8722ef187e4cd8ae90dab405235d4b 2023-03-20 10:58:22,672 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/6f8722ef187e4cd8ae90dab405235d4b, entries=150, sequenceid=144, filesize=11.8 K 2023-03-20 10:58:22,674 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/B/f82a203e14e94decb2bc0ee06b14cacb as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/f82a203e14e94decb2bc0ee06b14cacb 2023-03-20 10:58:22,678 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/f82a203e14e94decb2bc0ee06b14cacb, entries=150, sequenceid=144, filesize=11.8 K 2023-03-20 10:58:22,679 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/C/c0be06edbe84449fb269f3e4e3c2bb88 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/c0be06edbe84449fb269f3e4e3c2bb88 2023-03-20 10:58:22,683 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/c0be06edbe84449fb269f3e4e3c2bb88, entries=150, sequenceid=144, filesize=11.8 K 2023-03-20 10:58:22,684 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~386.11 KB/395376, currentSize=0 B/0 for 00ef1cabe628ad083ccaa287ccbbfb01 in 70ms, sequenceid=144, compaction requested=true 2023-03-20 10:58:22,684 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 00ef1cabe628ad083ccaa287ccbbfb01: 2023-03-20 10:58:22,684 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:22,684 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:22,684 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:22,684 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,684 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:22,685 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:22,686 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:22,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:22,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:22,686 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:22,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:22,687 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:22,687 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:22,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:22,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:22,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:22,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f7af36b[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:22,690 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:22,690 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:22,691 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,692 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:22,692 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:22,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:22,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:22,692 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:22,692 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:22,692 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:22,694 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:22,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:22,694 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:22,694 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:22,694 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:22,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:22,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:22,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:22,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:22,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:22,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:22,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:22,696 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:22,696 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:22,696 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:22,696 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:22,696 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:22,698 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:22,698 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:22,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:22,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:22,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:22,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:22,699 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:22,699 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:22,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:22,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:22,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:22,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:22,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:22,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:22,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:22,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:22,704 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:22,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,711 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,711 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:22,711 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:22,711 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,711 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:22,711 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:22,711 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:22,711 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:22,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:22,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:22,711 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:22,711 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:22,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:22,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:22,711 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:22,712 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:22,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:22,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:22,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 00ef1cabe628ad083ccaa287ccbbfb01 2023-03-20 10:58:22,784 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 00ef1cabe628ad083ccaa287ccbbfb01 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-20 10:58:22,785 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=A 2023-03-20 10:58:22,785 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:22,785 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=B 2023-03-20 10:58:22,785 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:22,786 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=C 2023-03-20 10:58:22,786 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:22,812 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:22,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:22,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:22,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:22,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:22,826 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:22,826 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:22,826 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:22,826 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:22,826 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:22,826 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:22,827 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:22,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:22,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:22,827 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,828 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,828 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:22,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:22,829 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:22,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:22,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:22,829 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:22,829 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:22,829 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:22,829 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:22,829 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:22,830 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:22,830 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:22,830 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:22,830 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:22,830 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:22,830 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. as already flushing 2023-03-20 10:58:22,830 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:22,830 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:22,830 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:22,830 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,830 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:22,830 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:22,831 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,831 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:22,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:22,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:22,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:22,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:22,833 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:22,833 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:22,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:22,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:22,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:22,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e2c5c20[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:22,834 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:22,834 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:22,835 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:22,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:22,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:22,835 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:22,835 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:22,835 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:22,835 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,835 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:22,836 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,836 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:22,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:22,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:22,836 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:22,836 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:22,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:22,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:22,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:22,838 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1679309902822, max timestamp=1679309902836Num uniques 859; 2023-03-20 10:58:22,838 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-20 10:58:22,839 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679309902822, max timestamp=1679309902836Num uniques 859; 2023-03-20 10:58:22,839 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679309902822, max timestamp=1679309902836Num uniques 859; 2023-03-20 10:58:22,839 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-20 10:58:22,839 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-20 10:58:22,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:22,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:22,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:22,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:22,840 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:22,840 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:22,840 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:22,840 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:22,840 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:22,843 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:22,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:22,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:22,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:22,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:22,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:22,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:22,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:22,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:22,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:22,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,846 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:22,846 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:22,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:22,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:22,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:22,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:22,847 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:22,847 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,853 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:22,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:22,853 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:22,853 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:22,854 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:22,854 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:22,854 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:22,854 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,854 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:22,854 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:22,854 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:22,854 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:22,854 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:22,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:22,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:22,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:22,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:22,899 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:22,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52176 deadline: 1679309962879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,899 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:22,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52180 deadline: 1679309962880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,902 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:22,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52164 deadline: 1679309962886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,903 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:22,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52190 deadline: 1679309962887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,954 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:22,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:22,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:22,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:22,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:22,957 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:22,957 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:22,957 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:22,957 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:22,957 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:22,957 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:22,958 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,958 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:22,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:22,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:22,958 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,958 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:22,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:22,959 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:22,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:22,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:22,959 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:22,959 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:22,959 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:22,959 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:22,959 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:22,959 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:22,959 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:22,959 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:22,959 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:22,960 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:22,960 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. as already flushing 2023-03-20 10:58:22,960 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:22,960 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:22,960 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:22,960 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,960 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:22,960 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:22,961 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,961 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:22,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:22,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:22,961 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:22,961 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:22,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:22,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:22,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:22,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,963 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/B/e6f5408f90e34443b17c0cb00c201ee2 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/e6f5408f90e34443b17c0cb00c201ee2 2023-03-20 10:58:22,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:22,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:22,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54e4d555[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:22,966 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:22,966 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:22,967 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:22,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:22,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:22,967 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:22,967 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:22,967 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:22,968 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,968 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:22,968 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:22,968 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,968 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:22,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:22,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:22,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:22,968 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:22,969 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 00ef1cabe628ad083ccaa287ccbbfb01/B of 00ef1cabe628ad083ccaa287ccbbfb01 into e6f5408f90e34443b17c0cb00c201ee2(size=11.8 K), total size for store is 23.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:58:22,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:22,969 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 00ef1cabe628ad083ccaa287ccbbfb01: 2023-03-20 10:58:22,969 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01., storeName=00ef1cabe628ad083ccaa287ccbbfb01/B, priority=9, startTime=1679309902527; duration=0sec 2023-03-20 10:58:22,969 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:58:22,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:22,969 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-03-20 10:58:22,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:22,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:22,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:22,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:22,971 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:22,971 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:22,971 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:22,971 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:22,971 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:22,979 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:22,979 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:22,979 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:22,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:22,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:22,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:22,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:22,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:22,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:22,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:22,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:22,982 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 90550 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-03-20 10:58:22,982 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,982 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): 00ef1cabe628ad083ccaa287ccbbfb01/C is initiating minor compaction (all files) 2023-03-20 10:58:22,982 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 00ef1cabe628ad083ccaa287ccbbfb01/C in TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:22,983 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/4cd7075d7aff49879958571076203591, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/c4522be77b084496852c128262f434d9, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/ca97ba0489744cfe8516c598d74ddddc, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/c63dc7ef0cf14aeb999cbc9bf5f53846, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/04b8974f07564d7a8c6c9150497ab5a4, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/bc20d1eeffd3454388980049639db488, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/916e137f601944b5a5930ad568f2aecb, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/c0be06edbe84449fb269f3e4e3c2bb88] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp, totalSize=88.4 K 2023-03-20 10:58:22,985 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 4cd7075d7aff49879958571076203591, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1679309896734 2023-03-20 10:58:22,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:22,986 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting c4522be77b084496852c128262f434d9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1679309896746 2023-03-20 10:58:22,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:22,987 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,987 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting ca97ba0489744cfe8516c598d74ddddc, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1679309897377 2023-03-20 10:58:22,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:22,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:22,990 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting c63dc7ef0cf14aeb999cbc9bf5f53846, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=65, earliestPutTs=1679309898022 2023-03-20 10:58:22,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:22,991 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,991 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:22,991 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:22,991 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:22,991 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:22,991 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:22,991 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:22,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:22,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:22,991 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:22,992 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:22,992 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:22,992 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:22,992 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:22,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:22,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:22,992 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:22,992 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 04b8974f07564d7a8c6c9150497ab5a4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=78, earliestPutTs=1679309898659 2023-03-20 10:58:22,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:22,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:22,993 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting bc20d1eeffd3454388980049639db488, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1679309899849 2023-03-20 10:58:22,993 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 916e137f601944b5a5930ad568f2aecb, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=110, earliestPutTs=1679309902027 2023-03-20 10:58:22,996 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting c0be06edbe84449fb269f3e4e3c2bb88, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1679309902127 2023-03-20 10:58:23,003 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/A/4388784eb4b0421f970250da1e392842 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/4388784eb4b0421f970250da1e392842 2023-03-20 10:58:23,007 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 00ef1cabe628ad083ccaa287ccbbfb01/A of 00ef1cabe628ad083ccaa287ccbbfb01 into 4388784eb4b0421f970250da1e392842(size=11.8 K), total size for store is 23.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:58:23,007 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 00ef1cabe628ad083ccaa287ccbbfb01: 2023-03-20 10:58:23,007 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01., storeName=00ef1cabe628ad083ccaa287ccbbfb01/A, priority=9, startTime=1679309902527; duration=0sec 2023-03-20 10:58:23,007 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:58:23,012 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): 00ef1cabe628ad083ccaa287ccbbfb01#C#compaction#410 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:58:23,015 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:23,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52180 deadline: 1679309963000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,015 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:23,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52176 deadline: 1679309963000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,015 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:23,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52164 deadline: 1679309963004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,015 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:23,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52190 deadline: 1679309963004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,092 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:23,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:23,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:23,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:23,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:23,094 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:23,094 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:23,094 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:23,094 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:23,095 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:23,095 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:23,095 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,095 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:23,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:23,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:23,096 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,096 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:23,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:23,096 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:23,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:23,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:23,096 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:23,096 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:23,096 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:23,096 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:23,096 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:23,096 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:23,097 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:23,097 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:23,097 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:23,097 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:23,097 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. as already flushing 2023-03-20 10:58:23,097 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:23,097 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:23,097 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:23,097 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,097 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:23,097 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:23,098 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,098 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:23,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:23,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:23,098 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:23,098 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:23,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:23,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:23,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:23,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:23,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:23,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@609189c8[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:23,099 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:23,099 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:23,099 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:23,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:23,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:23,100 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:23,100 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:23,100 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:23,100 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,100 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:23,100 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:23,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:23,100 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:23,100 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:23,101 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:23,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:23,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:23,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:23,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:23,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:23,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:23,104 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:23,104 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:23,104 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:23,104 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:23,104 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:23,104 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:23,105 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:23,105 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:23,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:23,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:23,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:23,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:23,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:23,105 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:23,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:23,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:23,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:23,106 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:23,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:23,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:23,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:23,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,107 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:23,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:23,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,108 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:23,109 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:23,109 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:23,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:23,109 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:23,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:23,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:23,109 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:23,109 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:23,109 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,109 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:23,109 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:23,109 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:23,109 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:23,109 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:23,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:23,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:23,204 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=155 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/A/3ef59604ecbf4790a08011bd978a5ccc 2023-03-20 10:58:23,210 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:23,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:23,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:23,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:23,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:23,222 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:23,222 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:23,222 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:23,222 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:23,223 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:23,223 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:23,223 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:23,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52180 deadline: 1679309963216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,223 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:23,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52164 deadline: 1679309963217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,224 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:23,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:23,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:23,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:23,225 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:23,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52190 deadline: 1679309963218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,225 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,225 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:23,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52176 deadline: 1679309963219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,225 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:23,225 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,226 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:23,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:23,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:23,226 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:23,230 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:23,230 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:23,230 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:23,230 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:23,230 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:23,230 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:23,230 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=155 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/B/ee5ae351142243ff83418d436234a9d8 2023-03-20 10:58:23,230 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:23,230 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:23,230 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:23,231 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. as already flushing 2023-03-20 10:58:23,231 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:23,231 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:23,231 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:23,231 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,231 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:23,231 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:23,232 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:23,232 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:23,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:23,232 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:23,232 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:23,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:23,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:23,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:23,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:23,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:23,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@898cf1a[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:23,234 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:23,235 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:23,236 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:23,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:23,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:23,236 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:23,237 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,237 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:23,237 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:23,237 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:23,237 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,237 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:23,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:23,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:23,237 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:23,238 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:23,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:23,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:23,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:23,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:23,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:23,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:23,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:23,241 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:23,241 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:23,242 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:23,242 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:23,242 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:23,253 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:23,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:23,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:23,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:23,253 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:23,253 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:23,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:23,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:23,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:23,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:23,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:23,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:23,254 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:23,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:23,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:23,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,255 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:23,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:23,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,260 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:23,260 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:23,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:23,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:23,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:23,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:23,261 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:23,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:23,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:23,261 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:23,261 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:23,263 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,263 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:23,263 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:23,263 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,263 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:23,263 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:23,263 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:23,362 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:23,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:23,364 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:23,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:23,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:23,369 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:23,369 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:23,369 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:23,370 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:23,370 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:23,370 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:23,371 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:23,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:23,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:23,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:23,371 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,372 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,372 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:23,372 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:23,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:23,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:23,372 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:23,378 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:23,378 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:23,378 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:23,378 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:23,378 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:23,390 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:23,390 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:23,402 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:23,402 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:23,402 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. as already flushing 2023-03-20 10:58:23,402 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:23,402 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:23,402 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:23,402 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,402 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:23,402 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:23,404 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,404 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:23,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:23,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:23,404 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:23,404 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:23,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:23,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:23,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:23,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:23,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:23,405 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14c94f75[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:23,405 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:23,405 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:23,406 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:23,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:23,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:23,406 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:23,406 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:23,406 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:23,406 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,406 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:23,407 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,407 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:23,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:23,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:23,407 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:23,407 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:23,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:23,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:23,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:23,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:23,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:23,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:23,409 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:23,409 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:23,409 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:23,409 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:23,409 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:23,409 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:23,410 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:23,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:23,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:23,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:23,410 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:23,410 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:23,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:23,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:23,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:23,410 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:23,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:23,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:23,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:23,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:23,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:23,411 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:23,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:23,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,413 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,413 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:23,413 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:23,413 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:23,413 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:23,413 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:23,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:23,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:23,413 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:23,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:23,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:23,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:23,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:23,413 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,414 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:23,414 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:23,414 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:23,414 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:23,414 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:23,428 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/C/21dd6e07cfe14bf4a78ac7e660acb83f as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/21dd6e07cfe14bf4a78ac7e660acb83f 2023-03-20 10:58:23,436 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 00ef1cabe628ad083ccaa287ccbbfb01/C of 00ef1cabe628ad083ccaa287ccbbfb01 into 21dd6e07cfe14bf4a78ac7e660acb83f(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:58:23,436 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 00ef1cabe628ad083ccaa287ccbbfb01: 2023-03-20 10:58:23,436 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01., storeName=00ef1cabe628ad083ccaa287ccbbfb01/C, priority=8, startTime=1679309902527; duration=0sec 2023-03-20 10:58:23,436 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:58:23,514 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:23,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:23,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:23,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:23,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:23,517 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:23,517 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:23,517 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:23,517 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:23,517 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:23,517 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:23,518 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,518 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:23,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:23,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:23,518 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,518 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:23,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:23,519 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:23,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:23,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:23,519 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:23,519 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:23,519 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:23,519 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:23,519 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:23,520 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:23,533 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:23,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52164 deadline: 1679309963524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,534 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:23,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52180 deadline: 1679309963526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,535 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:23,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52190 deadline: 1679309963526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,535 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:23,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52176 deadline: 1679309963526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,538 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:23,538 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:23,538 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:23,538 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:23,538 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. as already flushing 2023-03-20 10:58:23,538 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:23,538 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:23,538 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:23,538 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,538 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:23,538 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:23,539 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,540 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:23,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:23,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:23,540 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:23,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:23,540 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:23,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:23,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:23,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:23,545 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:23,545 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66757ec4[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:23,545 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:23,545 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:23,555 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,555 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:23,555 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:23,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:23,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:23,556 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:23,556 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:23,556 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:23,559 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,559 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:23,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:23,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:23,559 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:23,559 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:23,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:23,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:23,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:23,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:23,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:23,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:23,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:23,561 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:23,561 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:23,561 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:23,561 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:23,561 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:23,562 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:23,563 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:23,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:23,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:23,563 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,563 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:23,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:23,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:23,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:23,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:23,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:23,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:23,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:23,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:23,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:23,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:23,566 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:23,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:23,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,568 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:23,568 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:23,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:23,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:23,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:23,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:23,569 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,569 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:23,569 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:23,569 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,569 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:23,569 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:23,569 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:23,569 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:23,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:23,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:23,570 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:23,570 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:23,571 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:23,670 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:23,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:23,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:23,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:23,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:23,672 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:23,672 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:23,672 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:23,672 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:23,673 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:23,673 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:23,673 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:23,673 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:23,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:23,674 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,674 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:23,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:23,674 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:23,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:23,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:23,674 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:23,674 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:23,674 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:23,674 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:23,675 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:23,675 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:23,675 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:23,675 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:23,675 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:23,675 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:23,675 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. as already flushing 2023-03-20 10:58:23,675 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:23,675 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:23,675 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:23,675 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,675 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:23,675 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:23,677 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=155 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/C/fd82ec6f3dc74f609ce51093b779cd32 2023-03-20 10:58:23,686 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,686 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:23,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:23,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:23,686 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:23,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:23,686 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:23,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:23,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:23,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:23,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:23,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a46740b[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:23,688 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:23,688 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:23,688 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/A/3ef59604ecbf4790a08011bd978a5ccc as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/3ef59604ecbf4790a08011bd978a5ccc 2023-03-20 10:58:23,693 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/3ef59604ecbf4790a08011bd978a5ccc, entries=250, sequenceid=155, filesize=16.4 K 2023-03-20 10:58:23,694 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:23,694 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/B/ee5ae351142243ff83418d436234a9d8 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/ee5ae351142243ff83418d436234a9d8 2023-03-20 10:58:23,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:23,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:23,694 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:23,695 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:23,695 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:23,695 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,695 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:23,699 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/ee5ae351142243ff83418d436234a9d8, entries=150, sequenceid=155, filesize=11.8 K 2023-03-20 10:58:23,699 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/C/fd82ec6f3dc74f609ce51093b779cd32 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/fd82ec6f3dc74f609ce51093b779cd32 2023-03-20 10:58:23,703 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:23,703 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:23,703 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:23,703 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:23,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:23,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:23,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:23,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:23,704 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/fd82ec6f3dc74f609ce51093b779cd32, entries=150, sequenceid=155, filesize=11.8 K 2023-03-20 10:58:23,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:23,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:23,704 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 00ef1cabe628ad083ccaa287ccbbfb01 in 920ms, sequenceid=155, compaction requested=true 2023-03-20 10:58:23,705 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 00ef1cabe628ad083ccaa287ccbbfb01: 2023-03-20 10:58:23,705 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:58:23,705 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 10:58:23,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,705 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 10:58:23,705 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:58:23,705 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:58:23,705 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40978 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 10:58:23,706 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36198 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 10:58:23,706 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1912): 00ef1cabe628ad083ccaa287ccbbfb01/A is initiating minor compaction (all files) 2023-03-20 10:58:23,706 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): 00ef1cabe628ad083ccaa287ccbbfb01/B is initiating minor compaction (all files) 2023-03-20 10:58:23,706 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 00ef1cabe628ad083ccaa287ccbbfb01/A in TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:23,706 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 00ef1cabe628ad083ccaa287ccbbfb01/B in TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:23,706 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/e6f5408f90e34443b17c0cb00c201ee2, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/f82a203e14e94decb2bc0ee06b14cacb, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/ee5ae351142243ff83418d436234a9d8] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp, totalSize=35.3 K 2023-03-20 10:58:23,706 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/4388784eb4b0421f970250da1e392842, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/6f8722ef187e4cd8ae90dab405235d4b, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/3ef59604ecbf4790a08011bd978a5ccc] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp, totalSize=40.0 K 2023-03-20 10:58:23,706 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting e6f5408f90e34443b17c0cb00c201ee2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=110, earliestPutTs=1679309899862 2023-03-20 10:58:23,707 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 4388784eb4b0421f970250da1e392842, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=110, earliestPutTs=1679309899862 2023-03-20 10:58:23,707 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting f82a203e14e94decb2bc0ee06b14cacb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1679309902127 2023-03-20 10:58:23,707 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 6f8722ef187e4cd8ae90dab405235d4b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1679309902127 2023-03-20 10:58:23,707 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting ee5ae351142243ff83418d436234a9d8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=155, earliestPutTs=1679309902778 2023-03-20 10:58:23,708 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 3ef59604ecbf4790a08011bd978a5ccc, keycount=250, bloomtype=ROW, size=16.4 K, encoding=NONE, compression=NONE, seqNum=155, earliestPutTs=1679309902774 2023-03-20 10:58:23,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:23,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:23,710 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:23,710 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:23,710 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:23,710 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:23,710 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:23,711 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:23,711 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:23,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:23,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:23,711 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:23,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:23,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:23,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:23,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:23,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:23,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:23,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:23,713 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:23,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:23,714 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:23,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:23,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:23,718 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): 00ef1cabe628ad083ccaa287ccbbfb01#B#compaction#413 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:58:23,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,720 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:23,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:23,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:23,720 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:23,720 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:23,720 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] throttle.PressureAwareThroughputController(145): 00ef1cabe628ad083ccaa287ccbbfb01#A#compaction#414 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:58:23,721 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,721 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:23,721 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:23,721 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,721 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:23,721 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:23,721 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:23,721 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:23,721 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:23,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:23,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:23,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:23,722 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:23,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:23,739 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/B/7594663222dd445db928fd873cf75b47 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/7594663222dd445db928fd873cf75b47 2023-03-20 10:58:23,742 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 00ef1cabe628ad083ccaa287ccbbfb01/B of 00ef1cabe628ad083ccaa287ccbbfb01 into 7594663222dd445db928fd873cf75b47(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:58:23,742 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 00ef1cabe628ad083ccaa287ccbbfb01: 2023-03-20 10:58:23,742 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01., storeName=00ef1cabe628ad083ccaa287ccbbfb01/B, priority=13, startTime=1679309903705; duration=0sec 2023-03-20 10:58:23,743 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:58:23,743 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-20 10:58:23,743 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-20 10:58:23,743 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-20 10:58:23,743 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. because compaction request was cancelled 2023-03-20 10:58:23,820 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:23,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:23,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:23,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:23,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:23,823 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:23,823 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:23,823 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:23,823 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:23,823 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:23,823 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:23,824 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:23,824 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:23,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:23,824 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:23,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:23,824 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:23,824 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:23,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:23,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:23,824 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:23,825 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:23,825 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:23,825 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:23,825 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:23,825 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:23,830 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:23,830 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:23,830 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:23,830 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:23,830 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 00ef1cabe628ad083ccaa287ccbbfb01 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-03-20 10:58:23,830 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=A 2023-03-20 10:58:23,830 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:58:23,830 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=B 2023-03-20 10:58:23,831 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:58:23,831 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=C 2023-03-20 10:58:23,831 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:58:24,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 00ef1cabe628ad083ccaa287ccbbfb01 2023-03-20 10:58:24,038 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. as already flushing 2023-03-20 10:58:24,055 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:24,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52180 deadline: 1679309964047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:24,057 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:24,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52190 deadline: 1679309964047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:24,058 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:24,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52176 deadline: 1679309964055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:24,066 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:24,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52164 deadline: 1679309964056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:24,138 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/A/d1f0f3e8968a4e61bd90ccec27b410b7 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/d1f0f3e8968a4e61bd90ccec27b410b7 2023-03-20 10:58:24,141 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 00ef1cabe628ad083ccaa287ccbbfb01/A of 00ef1cabe628ad083ccaa287ccbbfb01 into d1f0f3e8968a4e61bd90ccec27b410b7(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:58:24,141 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 00ef1cabe628ad083ccaa287ccbbfb01: 2023-03-20 10:58:24,141 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01., storeName=00ef1cabe628ad083ccaa287ccbbfb01/A, priority=13, startTime=1679309903705; duration=0sec 2023-03-20 10:58:24,141 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:58:24,164 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:24,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52180 deadline: 1679309964156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:24,165 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:24,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52190 deadline: 1679309964158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:24,165 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:24,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52176 deadline: 1679309964159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:24,174 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:24,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52164 deadline: 1679309964167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:24,261 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=194 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/A/ee711ea0dc81455bb7a494f36fd5d371 2023-03-20 10:58:24,374 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:24,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52190 deadline: 1679309964366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:24,374 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:24,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52176 deadline: 1679309964366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:24,374 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:24,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52180 deadline: 1679309964366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:24,384 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:24,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52164 deadline: 1679309964377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:24,669 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=194 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/B/0d4cb271aaf54ce1b924a495aa98f312 2023-03-20 10:58:24,676 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=194 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/C/14937910ff8540c3b1bedc30281a2b37 2023-03-20 10:58:24,679 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:24,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52190 deadline: 1679309964675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:24,680 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/A/ee711ea0dc81455bb7a494f36fd5d371 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/ee711ea0dc81455bb7a494f36fd5d371 2023-03-20 10:58:24,680 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:24,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52176 deadline: 1679309964677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:24,681 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:24,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52180 deadline: 1679309964677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:24,683 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/ee711ea0dc81455bb7a494f36fd5d371, entries=150, sequenceid=194, filesize=11.8 K 2023-03-20 10:58:24,683 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/B/0d4cb271aaf54ce1b924a495aa98f312 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/0d4cb271aaf54ce1b924a495aa98f312 2023-03-20 10:58:24,686 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/0d4cb271aaf54ce1b924a495aa98f312, entries=150, sequenceid=194, filesize=11.8 K 2023-03-20 10:58:24,688 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/C/14937910ff8540c3b1bedc30281a2b37 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/14937910ff8540c3b1bedc30281a2b37 2023-03-20 10:58:24,690 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:24,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52164 deadline: 1679309964687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:24,692 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/14937910ff8540c3b1bedc30281a2b37, entries=150, sequenceid=194, filesize=11.8 K 2023-03-20 10:58:24,693 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for 00ef1cabe628ad083ccaa287ccbbfb01 in 863ms, sequenceid=194, compaction requested=true 2023-03-20 10:58:24,693 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 00ef1cabe628ad083ccaa287ccbbfb01: 2023-03-20 10:58:24,693 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:24,693 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:24,693 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:24,693 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:24,693 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:24,693 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:24,694 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:24,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:24,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:24,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:24,694 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:24,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:24,695 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:24,695 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:24,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:24,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:24,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:24,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:24,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:24,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44e88a46[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:24,696 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:24,696 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:24,697 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:24,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:24,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:24,697 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:24,697 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:24,697 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:24,697 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:24,697 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:24,698 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:24,698 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:24,698 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:24,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:24,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:24,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:24,698 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:24,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:24,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:24,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:24,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:24,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:24,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:24,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:24,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:24,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:24,700 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:24,700 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:24,700 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:24,701 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:24,701 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:24,701 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:24,702 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:24,702 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:24,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:24,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:24,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:24,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:24,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:24,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:24,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:24,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:24,703 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:24,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:24,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:24,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:24,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:24,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:24,704 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:24,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:24,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:24,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:24,706 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:24,706 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:24,706 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:24,706 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:24,706 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:24,706 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:24,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:24,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:24,706 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:24,706 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:24,706 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:24,706 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:24,707 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:24,707 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:24,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:24,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:24,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:24,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:24,807 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:24,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:24,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:24,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:24,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:24,810 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:24,810 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:24,810 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:24,810 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:24,810 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:24,810 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:24,811 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:24,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:24,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:24,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:24,812 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:24,812 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:24,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:24,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:24,813 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:24,813 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:24,813 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:24,813 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:24,813 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:24,813 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:24,813 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:24,813 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:24,818 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:24,818 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:24,818 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:24,818 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:24,818 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 00ef1cabe628ad083ccaa287ccbbfb01 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-20 10:58:24,818 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=A 2023-03-20 10:58:24,818 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:24,818 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=B 2023-03-20 10:58:24,818 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:24,819 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=C 2023-03-20 10:58:24,819 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:24,835 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=206 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/A/344e9944b1bc4a399c934b16eac07621 2023-03-20 10:58:25,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 00ef1cabe628ad083ccaa287ccbbfb01 2023-03-20 10:58:25,184 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. as already flushing 2023-03-20 10:58:25,212 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:25,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52164 deadline: 1679309965207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:25,212 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:25,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52176 deadline: 1679309965209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:25,213 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679309905204, max timestamp=1679309905211Num uniques 859; 2023-03-20 10:58:25,213 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-20 10:58:25,213 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679309905204, max timestamp=1679309905211Num uniques 859; 2023-03-20 10:58:25,213 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-20 10:58:25,214 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679309905204, max timestamp=1679309905211Num uniques 859; 2023-03-20 10:58:25,214 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-20 10:58:25,252 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=206 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/B/8c8906e63959467ca2abd8aafb6abe99 2023-03-20 10:58:25,263 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:25,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52180 deadline: 1679309965261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:25,269 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:25,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52190 deadline: 1679309965262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:25,317 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:25,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52164 deadline: 1679309965315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:25,317 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:25,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52176 deadline: 1679309965315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:25,369 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:25,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52180 deadline: 1679309965364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:25,375 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:25,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52190 deadline: 1679309965371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:25,522 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:25,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52176 deadline: 1679309965519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:25,523 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:25,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52164 deadline: 1679309965519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:25,580 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:25,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52180 deadline: 1679309965571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:25,582 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:25,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52190 deadline: 1679309965578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:25,664 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=206 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/C/81a11da5ee5e4871800279b4514864cf 2023-03-20 10:58:25,668 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/A/344e9944b1bc4a399c934b16eac07621 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/344e9944b1bc4a399c934b16eac07621 2023-03-20 10:58:25,673 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/344e9944b1bc4a399c934b16eac07621, entries=150, sequenceid=206, filesize=11.8 K 2023-03-20 10:58:25,674 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/B/8c8906e63959467ca2abd8aafb6abe99 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/8c8906e63959467ca2abd8aafb6abe99 2023-03-20 10:58:25,682 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/8c8906e63959467ca2abd8aafb6abe99, entries=150, sequenceid=206, filesize=11.8 K 2023-03-20 10:58:25,682 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/C/81a11da5ee5e4871800279b4514864cf as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/81a11da5ee5e4871800279b4514864cf 2023-03-20 10:58:25,688 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/81a11da5ee5e4871800279b4514864cf, entries=150, sequenceid=206, filesize=11.8 K 2023-03-20 10:58:25,688 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=129.71 KB/132820 for 00ef1cabe628ad083ccaa287ccbbfb01 in 870ms, sequenceid=206, compaction requested=true 2023-03-20 10:58:25,688 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 00ef1cabe628ad083ccaa287ccbbfb01: 2023-03-20 10:58:25,688 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:25,688 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:25,688 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:25,689 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:25,689 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:25,689 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:25,777 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:25,778 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:25,778 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:25,778 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:25,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:25,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:25,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:25,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:25,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:25,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:25,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:25,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:25,784 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:25,784 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a04d749[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:25,784 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:25,784 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:25,787 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:25,787 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:25,787 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:25,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:25,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:25,787 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:25,787 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:25,787 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:25,789 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:25,789 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:25,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:25,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:25,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:25,789 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:25,789 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:25,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:25,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:25,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:25,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:25,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:25,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:25,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:25,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:25,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:25,793 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:25,793 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:25,793 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:25,793 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:25,793 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:25,795 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:25,795 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:25,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:25,795 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:25,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:25,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:25,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:25,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:25,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:25,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:25,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:25,796 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:25,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:25,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:25,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:25,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:25,796 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:25,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:25,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:25,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:25,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:25,800 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:25,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:25,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:25,800 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:25,800 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:25,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:25,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:25,801 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:25,801 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:25,801 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:25,801 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:25,801 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:25,801 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:25,801 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:25,801 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:25,801 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:25,801 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:25,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:25,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:25,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 00ef1cabe628ad083ccaa287ccbbfb01 2023-03-20 10:58:25,833 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 00ef1cabe628ad083ccaa287ccbbfb01 3/3 column families, dataSize=136.42 KB heapSize=385.38 KB 2023-03-20 10:58:25,834 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=A 2023-03-20 10:58:25,834 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:58:25,834 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=B 2023-03-20 10:58:25,834 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:58:25,834 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=C 2023-03-20 10:58:25,834 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:58:25,843 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=242 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/A/e2950c7a878747c38c81394aaf5cc938 2023-03-20 10:58:25,868 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:25,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52176 deadline: 1679309965858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:25,877 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:25,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52164 deadline: 1679309965868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:25,890 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:25,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52190 deadline: 1679309965884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:25,895 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:25,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52180 deadline: 1679309965885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:25,901 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:25,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:25,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:25,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:25,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:25,903 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:25,903 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:25,903 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:25,903 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:25,904 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:25,904 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:25,904 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:25,904 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:25,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:25,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:25,905 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:25,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:25,905 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:25,905 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:25,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:25,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:25,905 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:25,906 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:25,906 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:25,906 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:25,906 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:25,906 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:25,906 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:25,906 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:25,906 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:25,906 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:25,906 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. as already flushing 2023-03-20 10:58:25,906 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:25,906 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:25,906 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:25,906 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:25,906 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:25,906 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:25,907 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:25,907 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:25,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:25,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:25,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:25,908 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:25,908 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:25,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:25,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:25,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:25,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:25,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:25,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:25,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57ca0993[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:25,909 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:25,909 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:25,909 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:25,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:25,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:25,909 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:25,909 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:25,909 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:25,910 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:25,910 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:25,910 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:25,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:25,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:25,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:25,910 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:25,910 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:25,910 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:25,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:25,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:25,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:25,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:25,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:25,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:25,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:25,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:25,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:25,912 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:25,912 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:25,912 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:25,912 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:25,912 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:25,913 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:25,913 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:25,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:25,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:25,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:25,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:25,913 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:25,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:25,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:25,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:25,913 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:25,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:25,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:25,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:25,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:25,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:25,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:25,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:25,915 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:25,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:25,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:25,916 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:25,916 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:25,916 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:25,916 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:25,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:25,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:25,916 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:25,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:25,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:25,916 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:25,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:25,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:25,916 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:25,916 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:25,916 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:25,916 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:25,916 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:25,916 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:25,981 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:25,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52176 deadline: 1679309965971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:25,985 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:25,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52164 deadline: 1679309965981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,016 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:26,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:26,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:26,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:26,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:26,019 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:26,019 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:26,019 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:26,019 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:26,019 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:26,019 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:26,020 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,020 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,020 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:26,020 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:26,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:26,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:26,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:26,021 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:26,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:26,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:26,021 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:26,021 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:26,021 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:26,022 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:26,022 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:26,022 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:26,026 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:26,026 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:26,026 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:26,027 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:26,027 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. as already flushing 2023-03-20 10:58:26,027 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:26,027 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:26,027 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:26,027 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,027 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:26,027 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:26,037 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,037 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:26,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:26,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:26,037 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:26,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:26,037 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:26,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:26,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:26,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:26,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:26,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e15c94d[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:26,039 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:26,039 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:26,040 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:26,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:26,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:26,040 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:26,040 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:26,040 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:26,040 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,040 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:26,041 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,041 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:26,041 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:26,041 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:26,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:26,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:26,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:26,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:26,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:26,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:26,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:26,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:26,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:26,044 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:26,044 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:26,044 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:26,044 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:26,044 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:26,045 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:26,045 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:26,045 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:26,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:26,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:26,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:26,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:26,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:26,046 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:26,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:26,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:26,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:26,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:26,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:26,047 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:26,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:26,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:26,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,048 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,048 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:26,048 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:26,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:26,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:26,048 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:26,048 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:26,048 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,048 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:26,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:26,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:26,048 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:26,048 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:26,048 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:26,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:26,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:26,049 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:26,048 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:26,049 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:26,149 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:26,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:26,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:26,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:26,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:26,154 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:26,154 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:26,154 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:26,154 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:26,155 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:26,155 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:26,155 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:26,155 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:26,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:26,156 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,156 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:26,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:26,156 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:26,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:26,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:26,156 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:26,156 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:26,156 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:26,156 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:26,157 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:26,157 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:26,157 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:26,157 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:26,157 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:26,157 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:26,157 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. as already flushing 2023-03-20 10:58:26,157 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:26,157 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:26,157 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:26,157 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,157 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:26,157 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:26,158 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,158 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:26,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:26,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:26,158 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:26,158 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:26,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:26,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:26,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:26,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:26,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:26,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6eafc473[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:26,160 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:26,160 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:26,161 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,161 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:26,161 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:26,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:26,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:26,161 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:26,161 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:26,161 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:26,162 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,162 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:26,162 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:26,162 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:26,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:26,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:26,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:26,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:26,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:26,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:26,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:26,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:26,164 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:26,164 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:26,164 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:26,164 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:26,164 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:26,164 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:26,165 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:26,165 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:26,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:26,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:26,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:26,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:26,165 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:26,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:26,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:26,166 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:26,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:26,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:26,167 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,168 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,168 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:26,168 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:26,168 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,168 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:26,168 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:26,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:26,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:26,168 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:26,168 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:26,168 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:26,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:26,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:26,168 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:26,169 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:26,169 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:26,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:26,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:26,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:26,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:26,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:26,171 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:26,188 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:26,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52176 deadline: 1679309966183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,197 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:26,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52164 deadline: 1679309966187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,257 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:26,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52136 deadline: 1679309966248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,261 DEBUG [Thread-2963] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8231 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01., hostname=jenkins-hbase20.apache.org,36041,1679309777062, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-20 10:58:26,269 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:26,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:26,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:26,278 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=242 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/B/c254b8df9d724d40985885924460caee 2023-03-20 10:58:26,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:26,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:26,286 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:26,286 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:26,286 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:26,286 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:26,287 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:26,287 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:26,287 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:26,287 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:26,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:26,289 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,289 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:26,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:26,289 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:26,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:26,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:26,289 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:26,290 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:26,290 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:26,290 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:26,290 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:26,290 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:26,290 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:26,290 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:26,290 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:26,290 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:26,290 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. as already flushing 2023-03-20 10:58:26,291 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:26,291 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:26,291 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:26,291 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,291 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:26,291 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:26,292 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,292 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:26,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:26,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:26,292 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:26,292 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:26,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:26,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:26,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:26,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:26,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:26,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@522de870[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:26,305 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:26,305 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:26,306 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=242 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/C/3471aa96965a46b7aa43ac1b7ac2d4b7 2023-03-20 10:58:26,311 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/A/e2950c7a878747c38c81394aaf5cc938 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/e2950c7a878747c38c81394aaf5cc938 2023-03-20 10:58:26,311 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:26,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:26,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:26,311 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:26,311 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:26,311 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:26,311 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,311 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:26,312 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,312 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:26,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:26,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:26,312 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:26,312 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:26,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:26,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:26,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:26,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:26,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:26,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:26,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:26,316 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:26,316 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:26,316 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:26,316 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:26,316 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:26,318 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/e2950c7a878747c38c81394aaf5cc938, entries=200, sequenceid=242, filesize=14.1 K 2023-03-20 10:58:26,319 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/B/c254b8df9d724d40985885924460caee as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/c254b8df9d724d40985885924460caee 2023-03-20 10:58:26,320 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:26,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:26,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:26,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:26,320 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:26,320 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:26,320 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:26,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:26,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:26,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:26,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:26,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:26,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:26,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:26,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:26,325 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,326 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/c254b8df9d724d40985885924460caee, entries=150, sequenceid=242, filesize=11.8 K 2023-03-20 10:58:26,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:26,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:26,326 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/C/3471aa96965a46b7aa43ac1b7ac2d4b7 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/3471aa96965a46b7aa43ac1b7ac2d4b7 2023-03-20 10:58:26,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,329 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/3471aa96965a46b7aa43ac1b7ac2d4b7, entries=150, sequenceid=242, filesize=11.8 K 2023-03-20 10:58:26,329 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~136.42 KB/139690, heapSize ~385.33 KB/394576, currentSize=46.96 KB/48090 for 00ef1cabe628ad083ccaa287ccbbfb01 in 496ms, sequenceid=242, compaction requested=true 2023-03-20 10:58:26,330 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 00ef1cabe628ad083ccaa287ccbbfb01: 2023-03-20 10:58:26,330 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:58:26,330 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 10:58:26,330 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:58:26,330 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-20 10:58:26,330 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-20 10:58:26,331 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 50876 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 10:58:26,331 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1912): 00ef1cabe628ad083ccaa287ccbbfb01/A is initiating minor compaction (all files) 2023-03-20 10:58:26,331 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 00ef1cabe628ad083ccaa287ccbbfb01/A in TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:26,331 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/d1f0f3e8968a4e61bd90ccec27b410b7, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/ee711ea0dc81455bb7a494f36fd5d371, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/344e9944b1bc4a399c934b16eac07621, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/e2950c7a878747c38c81394aaf5cc938] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp, totalSize=49.7 K 2023-03-20 10:58:26,332 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60454 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-20 10:58:26,332 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): 00ef1cabe628ad083ccaa287ccbbfb01/C is initiating minor compaction (all files) 2023-03-20 10:58:26,332 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 00ef1cabe628ad083ccaa287ccbbfb01/C in TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:26,332 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/21dd6e07cfe14bf4a78ac7e660acb83f, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/fd82ec6f3dc74f609ce51093b779cd32, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/14937910ff8540c3b1bedc30281a2b37, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/81a11da5ee5e4871800279b4514864cf, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/3471aa96965a46b7aa43ac1b7ac2d4b7] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp, totalSize=59.0 K 2023-03-20 10:58:26,332 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 21dd6e07cfe14bf4a78ac7e660acb83f, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1679309902127 2023-03-20 10:58:26,333 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting fd82ec6f3dc74f609ce51093b779cd32, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=155, earliestPutTs=1679309902778 2023-03-20 10:58:26,333 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 14937910ff8540c3b1bedc30281a2b37, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=194, earliestPutTs=1679309902870 2023-03-20 10:58:26,334 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting d1f0f3e8968a4e61bd90ccec27b410b7, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=155, earliestPutTs=1679309902778 2023-03-20 10:58:26,334 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 81a11da5ee5e4871800279b4514864cf, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=206, earliestPutTs=1679309904044 2023-03-20 10:58:26,335 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting ee711ea0dc81455bb7a494f36fd5d371, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=194, earliestPutTs=1679309902870 2023-03-20 10:58:26,335 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 3471aa96965a46b7aa43ac1b7ac2d4b7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=242, earliestPutTs=1679309905235 2023-03-20 10:58:26,335 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 344e9944b1bc4a399c934b16eac07621, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=206, earliestPutTs=1679309904044 2023-03-20 10:58:26,335 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting e2950c7a878747c38c81394aaf5cc938, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=242, earliestPutTs=1679309905235 2023-03-20 10:58:26,336 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:26,336 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:26,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:26,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:26,336 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:26,336 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:26,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:26,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:26,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:26,336 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:26,337 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:26,337 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,337 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:26,337 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:26,337 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:26,337 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:26,337 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:26,337 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:26,359 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] throttle.PressureAwareThroughputController(145): 00ef1cabe628ad083ccaa287ccbbfb01#A#compaction#424 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:58:26,368 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): 00ef1cabe628ad083ccaa287ccbbfb01#C#compaction#425 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:58:26,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 00ef1cabe628ad083ccaa287ccbbfb01 2023-03-20 10:58:26,401 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 00ef1cabe628ad083ccaa287ccbbfb01 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-20 10:58:26,401 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=A 2023-03-20 10:58:26,401 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:26,401 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=B 2023-03-20 10:58:26,401 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:26,401 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=C 2023-03-20 10:58:26,401 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:26,412 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/A/c205c353a77f421e95819ebc08952346 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/c205c353a77f421e95819ebc08952346 2023-03-20 10:58:26,414 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/C/ce7e68b49a854fb5848a591dae576208 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/ce7e68b49a854fb5848a591dae576208 2023-03-20 10:58:26,420 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=253 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/A/974a9baf2d114c9aa5d90428dd5c5b13 2023-03-20 10:58:26,431 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 00ef1cabe628ad083ccaa287ccbbfb01/A of 00ef1cabe628ad083ccaa287ccbbfb01 into c205c353a77f421e95819ebc08952346(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:58:26,431 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 00ef1cabe628ad083ccaa287ccbbfb01: 2023-03-20 10:58:26,431 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01., storeName=00ef1cabe628ad083ccaa287ccbbfb01/A, priority=12, startTime=1679309906330; duration=0sec 2023-03-20 10:58:26,431 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:58:26,432 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 10:58:26,432 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 00ef1cabe628ad083ccaa287ccbbfb01/C of 00ef1cabe628ad083ccaa287ccbbfb01 into ce7e68b49a854fb5848a591dae576208(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:58:26,432 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 00ef1cabe628ad083ccaa287ccbbfb01: 2023-03-20 10:58:26,432 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01., storeName=00ef1cabe628ad083ccaa287ccbbfb01/C, priority=11, startTime=1679309906330; duration=0sec 2023-03-20 10:58:26,432 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:58:26,433 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48486 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 10:58:26,433 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1912): 00ef1cabe628ad083ccaa287ccbbfb01/B is initiating minor compaction (all files) 2023-03-20 10:58:26,433 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 00ef1cabe628ad083ccaa287ccbbfb01/B in TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:26,433 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/7594663222dd445db928fd873cf75b47, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/0d4cb271aaf54ce1b924a495aa98f312, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/8c8906e63959467ca2abd8aafb6abe99, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/c254b8df9d724d40985885924460caee] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp, totalSize=47.3 K 2023-03-20 10:58:26,433 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 7594663222dd445db928fd873cf75b47, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=155, earliestPutTs=1679309902778 2023-03-20 10:58:26,434 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 0d4cb271aaf54ce1b924a495aa98f312, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=194, earliestPutTs=1679309902870 2023-03-20 10:58:26,434 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 8c8906e63959467ca2abd8aafb6abe99, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=206, earliestPutTs=1679309904044 2023-03-20 10:58:26,434 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting c254b8df9d724d40985885924460caee, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=242, earliestPutTs=1679309905235 2023-03-20 10:58:26,437 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:26,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:26,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:26,456 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] throttle.PressureAwareThroughputController(145): 00ef1cabe628ad083ccaa287ccbbfb01#B#compaction#428 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:58:26,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:26,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:26,456 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:26,456 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:26,456 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:26,457 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:26,457 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:26,457 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:26,457 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:26,457 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:26,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:26,458 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:26,458 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:26,458 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:26,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:26,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:26,459 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:26,468 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:26,468 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:26,468 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:26,469 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:26,469 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:26,470 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:26,470 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:26,470 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:26,470 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:26,470 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. as already flushing 2023-03-20 10:58:26,470 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:26,470 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:26,470 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:26,470 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,470 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:26,470 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:26,471 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,471 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:26,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:26,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:26,472 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:26,472 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:26,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:26,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:26,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:26,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:26,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:26,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55acf537[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:26,473 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:26,473 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:26,475 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:26,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:26,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:26,476 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:26,476 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:26,476 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:26,476 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,476 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:26,477 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:26,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:26,477 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:26,477 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:26,477 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:26,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:26,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:26,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:26,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:26,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:26,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:26,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:26,481 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:26,481 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:26,481 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:26,481 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:26,481 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:26,482 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:26,482 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:26,483 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:26,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:26,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:26,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:26,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:26,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:26,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:26,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:26,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:26,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:26,484 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:26,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:26,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:26,486 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:26,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:26,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,487 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,488 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:26,488 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:26,488 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,488 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:26,488 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:26,488 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:26,488 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:26,488 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:26,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:26,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:26,488 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:26,488 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:26,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:26,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:26,488 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:26,488 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:26,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:26,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:26,498 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:26,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52180 deadline: 1679309966492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,499 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679309906468, max timestamp=1679309906497Num uniques 859; 2023-03-20 10:58:26,499 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-20 10:58:26,499 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679309906468, max timestamp=1679309906497Num uniques 859; 2023-03-20 10:58:26,499 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-20 10:58:26,499 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679309906468, max timestamp=1679309906497Num uniques 859; 2023-03-20 10:58:26,499 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-20 10:58:26,543 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:26,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52164 deadline: 1679309966534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,547 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:26,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52190 deadline: 1679309966538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,548 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:26,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52176 deadline: 1679309966543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,588 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:26,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:26,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:26,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:26,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:26,591 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:26,591 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:26,591 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:26,591 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:26,592 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:26,592 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:26,592 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:26,592 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:26,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:26,593 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:26,593 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:26,593 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:26,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:26,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:26,593 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:26,593 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:26,593 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:26,593 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:26,593 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:26,593 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:26,593 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:26,594 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:26,594 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:26,594 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:26,594 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. as already flushing 2023-03-20 10:58:26,594 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:26,594 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:26,594 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:26,594 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,594 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:26,594 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:26,595 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,595 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:26,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:26,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:26,595 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:26,595 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:26,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:26,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:26,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:26,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:26,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:26,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1533016e[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:26,596 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:26,596 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:26,597 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:26,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:26,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:26,597 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:26,597 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:26,597 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,597 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:26,597 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:26,598 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,598 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:26,598 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:26,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:26,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:26,598 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:26,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:26,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:26,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:26,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:26,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:26,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:26,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:26,600 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:26,600 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:26,600 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:26,600 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:26,600 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:26,601 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:26,601 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:26,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:26,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:26,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:26,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:26,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:26,601 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:26,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:26,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:26,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:26,602 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:26,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:26,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:26,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:26,603 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:26,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:26,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,604 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,604 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:26,604 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:26,604 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:26,604 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:26,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:26,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:26,604 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,604 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:26,604 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:26,604 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:26,604 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:26,604 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:26,604 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:26,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:26,604 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:26,605 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:26,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:26,606 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:26,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52180 deadline: 1679309966600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,652 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:26,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52164 deadline: 1679309966648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,652 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:26,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52190 deadline: 1679309966648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,653 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:26,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52176 deadline: 1679309966648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,705 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:26,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:26,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:26,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:26,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:26,707 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:26,707 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:26,707 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:26,707 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:26,707 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:26,707 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:26,708 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:26,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:26,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:26,708 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:26,708 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,708 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:26,708 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:26,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:26,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:26,709 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:26,709 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:26,709 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:26,709 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:26,709 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:26,709 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:26,709 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:26,709 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:26,709 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:26,709 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:26,709 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. as already flushing 2023-03-20 10:58:26,709 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:26,709 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:26,709 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:26,709 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,709 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:26,709 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:26,710 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,710 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:26,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:26,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:26,711 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:26,711 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:26,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:26,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:26,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:26,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:26,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:26,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5afc3e7[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:26,714 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:26,714 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:26,714 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:26,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:26,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:26,714 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:26,715 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:26,715 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:26,715 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,715 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:26,715 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:26,715 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,715 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:26,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:26,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:26,715 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:26,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:26,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:26,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:26,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:26,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:26,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:26,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:26,718 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:26,718 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:26,718 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:26,718 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:26,718 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:26,720 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:26,720 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:26,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:26,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:26,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:26,720 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:26,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:26,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:26,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:26,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:26,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:26,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:26,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:26,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:26,721 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:26,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:26,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:26,723 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,725 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,725 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:26,725 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:26,725 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,725 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:26,725 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:26,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:26,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:26,725 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:26,725 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:26,725 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:26,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:26,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:26,725 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:26,725 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:26,725 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:26,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:26,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:26,816 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:26,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52180 deadline: 1679309966809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,825 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:26,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:26,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:26,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:26,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:26,829 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:26,829 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:26,829 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:26,829 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:26,829 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:26,829 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:26,830 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,830 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:26,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:26,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:26,830 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,830 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:26,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:26,830 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:26,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:26,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:26,834 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:26,834 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:26,834 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:26,834 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:26,835 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:26,835 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:26,838 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:26,838 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:26,838 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:26,838 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:26,838 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. as already flushing 2023-03-20 10:58:26,838 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:26,838 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:26,838 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:26,838 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,838 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:26,838 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:26,844 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,844 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:26,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:26,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:26,845 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:26,845 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:26,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:26,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:26,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:26,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:26,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:26,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38fcd54a[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:26,846 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:26,846 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:26,846 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:26,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:26,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:26,846 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:26,846 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:26,846 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,847 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:26,846 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:26,847 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:26,847 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,847 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:26,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:26,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:26,847 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:26,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:26,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:26,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:26,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:26,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:26,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:26,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:26,849 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:26,849 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:26,849 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:26,849 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:26,849 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:26,851 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:26,852 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:26,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:26,852 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:26,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:26,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:26,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:26,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:26,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:26,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:26,852 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:26,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:26,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:26,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:26,857 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:26,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:26,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:26,860 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:26,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52164 deadline: 1679309966854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,860 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:26,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52176 deadline: 1679309966854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,861 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:26,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52190 deadline: 1679309966855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,861 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:26,861 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:26,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:26,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:26,861 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=253 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/B/8f08871fecc5470097849a1397920f86 2023-03-20 10:58:26,861 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,861 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:26,862 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:26,862 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,862 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:26,862 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:26,862 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:26,862 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:26,862 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:26,862 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:26,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:26,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:26,862 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:26,862 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:26,862 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:26,870 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=253 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/C/c6617e014ddf4699a5629c6f2e59b088 2023-03-20 10:58:26,873 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/A/974a9baf2d114c9aa5d90428dd5c5b13 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/974a9baf2d114c9aa5d90428dd5c5b13 2023-03-20 10:58:26,876 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/974a9baf2d114c9aa5d90428dd5c5b13, entries=150, sequenceid=253, filesize=11.8 K 2023-03-20 10:58:26,877 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/B/8f08871fecc5470097849a1397920f86 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/8f08871fecc5470097849a1397920f86 2023-03-20 10:58:26,879 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/8f08871fecc5470097849a1397920f86, entries=150, sequenceid=253, filesize=11.8 K 2023-03-20 10:58:26,880 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/C/c6617e014ddf4699a5629c6f2e59b088 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/c6617e014ddf4699a5629c6f2e59b088 2023-03-20 10:58:26,883 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/c6617e014ddf4699a5629c6f2e59b088, entries=150, sequenceid=253, filesize=11.8 K 2023-03-20 10:58:26,884 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/B/4356b31834ab4ea99e4c06f88b9c8a31 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/4356b31834ab4ea99e4c06f88b9c8a31 2023-03-20 10:58:26,885 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 00ef1cabe628ad083ccaa287ccbbfb01 in 483ms, sequenceid=253, compaction requested=false 2023-03-20 10:58:26,885 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 00ef1cabe628ad083ccaa287ccbbfb01: 2023-03-20 10:58:26,887 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 00ef1cabe628ad083ccaa287ccbbfb01/B of 00ef1cabe628ad083ccaa287ccbbfb01 into 4356b31834ab4ea99e4c06f88b9c8a31(size=12.2 K), total size for store is 24.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:58:26,887 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 00ef1cabe628ad083ccaa287ccbbfb01: 2023-03-20 10:58:26,887 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01., storeName=00ef1cabe628ad083ccaa287ccbbfb01/B, priority=12, startTime=1679309906330; duration=0sec 2023-03-20 10:58:26,887 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:58:26,962 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:26,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:26,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:26,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:26,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:26,967 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:26,967 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:26,967 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:26,967 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:26,967 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:26,967 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:26,968 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,968 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:26,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:26,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:26,968 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:26,968 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:26,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:26,968 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:26,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:26,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:26,969 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:26,969 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:26,969 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:26,969 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:26,969 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:26,969 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:26,969 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:26,969 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:26,969 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:26,969 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:26,969 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 00ef1cabe628ad083ccaa287ccbbfb01 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-03-20 10:58:26,969 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=A 2023-03-20 10:58:26,970 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:58:26,970 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=B 2023-03-20 10:58:26,970 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:58:26,970 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=C 2023-03-20 10:58:26,970 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:58:27,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 00ef1cabe628ad083ccaa287ccbbfb01 2023-03-20 10:58:27,122 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. as already flushing 2023-03-20 10:58:27,169 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:27,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52164 deadline: 1679309967161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,170 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:27,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52176 deadline: 1679309967162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,170 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:27,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52190 deadline: 1679309967164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,174 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:27,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52180 deadline: 1679309967169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,280 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:27,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52180 deadline: 1679309967275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,384 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=292 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/A/af065d914f91473a913141b8f5953e25 2023-03-20 10:58:27,399 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=292 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/B/e2d1d7ce6a8f4af99314b1d8d1f4f6e2 2023-03-20 10:58:27,410 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=292 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/C/e189b48dcee64e089821662663d83bc7 2023-03-20 10:58:27,414 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/A/af065d914f91473a913141b8f5953e25 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/af065d914f91473a913141b8f5953e25 2023-03-20 10:58:27,418 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/af065d914f91473a913141b8f5953e25, entries=150, sequenceid=292, filesize=11.9 K 2023-03-20 10:58:27,419 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/B/e2d1d7ce6a8f4af99314b1d8d1f4f6e2 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/e2d1d7ce6a8f4af99314b1d8d1f4f6e2 2023-03-20 10:58:27,427 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/e2d1d7ce6a8f4af99314b1d8d1f4f6e2, entries=150, sequenceid=292, filesize=11.9 K 2023-03-20 10:58:27,428 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/C/e189b48dcee64e089821662663d83bc7 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/e189b48dcee64e089821662663d83bc7 2023-03-20 10:58:27,431 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/e189b48dcee64e089821662663d83bc7, entries=150, sequenceid=292, filesize=11.9 K 2023-03-20 10:58:27,432 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 00ef1cabe628ad083ccaa287ccbbfb01 in 463ms, sequenceid=292, compaction requested=true 2023-03-20 10:58:27,432 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 00ef1cabe628ad083ccaa287ccbbfb01: 2023-03-20 10:58:27,432 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:27,432 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:27,432 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:27,432 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,438 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:27,438 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:27,439 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:27,440 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:27,440 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:27,442 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:27,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:27,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:27,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:27,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:27,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:27,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:27,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@acb8a2f[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:27,444 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:27,444 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:27,446 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:27,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:27,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:27,446 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:27,446 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,446 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:27,446 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:27,449 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:27,455 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:27,455 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,455 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:27,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:27,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:27,455 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:27,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:27,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:27,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:27,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:27,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:27,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:27,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:27,461 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:27,461 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:27,461 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:27,461 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:27,461 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:27,462 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:27,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:27,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:27,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:27,463 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:27,463 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:27,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:27,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:27,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:27,464 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:27,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:27,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:27,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:27,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:27,466 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:27,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:27,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:27,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,478 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:27,478 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:27,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:27,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:27,479 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:27,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:27,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:27,479 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,479 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:27,479 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:27,479 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,479 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:27,479 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:27,479 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:27,479 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:27,479 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:27,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:27,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:27,480 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:27,490 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 00ef1cabe628ad083ccaa287ccbbfb01 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-20 10:58:27,490 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=A 2023-03-20 10:58:27,490 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:27,490 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=B 2023-03-20 10:58:27,490 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:27,490 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=C 2023-03-20 10:58:27,490 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:27,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 00ef1cabe628ad083ccaa287ccbbfb01 2023-03-20 10:58:27,579 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:27,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:27,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:27,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:27,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:27,586 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:27,586 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:27,586 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:27,586 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:27,586 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:27,586 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:27,587 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:27,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:27,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:27,587 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:27,587 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,587 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:27,587 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:27,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:27,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:27,588 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:27,588 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:27,588 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:27,588 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:27,588 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:27,588 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:27,589 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:27,589 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:27,589 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:27,589 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:27,589 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. as already flushing 2023-03-20 10:58:27,589 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:27,589 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:27,590 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:27,590 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,590 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:27,590 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:27,592 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:27,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:27,592 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:27,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:27,592 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:27,592 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:27,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:27,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:27,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:27,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:27,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fed269[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:27,594 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:27,594 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:27,595 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:27,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:27,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:27,595 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:27,595 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:27,595 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,595 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:27,595 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:27,596 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,596 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:27,596 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:27,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:27,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:27,596 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:27,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:27,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:27,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:27,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:27,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:27,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:27,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:27,598 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:27,598 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:27,598 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:27,598 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:27,598 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:27,599 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:27,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:27,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:27,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:27,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:27,599 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:27,599 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:27,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:27,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:27,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:27,600 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:27,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:27,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:27,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:27,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:27,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:27,601 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:27,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,602 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:27,602 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:27,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:27,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:27,602 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:27,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:27,602 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:27,602 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:27,602 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:27,602 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,602 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:27,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:27,602 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:27,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:27,602 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:27,602 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:27,602 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:27,637 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:27,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52180 deadline: 1679309967631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,674 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:27,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52164 deadline: 1679309967671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,678 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:27,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52176 deadline: 1679309967674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,681 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:27,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52190 deadline: 1679309967675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,703 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:27,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:27,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:27,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:27,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:27,705 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:27,705 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:27,705 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:27,705 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:27,706 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:27,706 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:27,706 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:27,706 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:27,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:27,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:27,707 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,707 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:27,707 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:27,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:27,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:27,707 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:27,708 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:27,708 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:27,708 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:27,708 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:27,708 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:27,709 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:27,709 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:27,709 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:27,709 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:27,709 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. as already flushing 2023-03-20 10:58:27,709 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:27,709 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:27,709 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:27,709 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,709 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:27,709 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:27,710 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,710 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:27,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:27,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:27,711 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:27,711 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:27,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:27,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:27,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:27,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:27,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:27,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7cb5898a[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:27,712 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:27,712 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:27,712 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:27,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:27,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:27,713 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,713 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:27,713 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:27,713 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:27,713 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:27,715 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,715 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:27,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:27,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:27,715 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:27,715 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:27,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:27,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:27,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:27,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:27,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:27,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:27,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:27,718 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:27,718 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:27,718 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:27,718 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:27,718 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:27,719 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:27,719 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:27,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:27,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:27,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:27,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:27,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:27,719 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:27,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:27,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:27,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:27,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:27,725 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:27,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:27,725 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:27,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:27,737 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:27,737 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:27,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:27,737 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:27,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:27,737 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,737 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:27,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:27,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:27,737 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:27,737 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,737 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:27,738 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:27,738 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:27,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:27,738 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:27,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:27,738 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:27,738 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:27,742 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:27,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52180 deadline: 1679309967738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,839 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:27,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:27,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:27,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:27,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:27,845 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:27,846 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:27,846 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:27,846 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:27,846 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:27,846 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:27,847 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:27,847 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:27,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:27,847 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,847 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:27,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:27,847 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:27,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:27,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:27,848 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:27,848 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:27,848 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:27,848 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:27,848 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:27,848 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:27,857 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:27,857 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:27,857 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:27,857 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:27,857 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. as already flushing 2023-03-20 10:58:27,857 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:27,857 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:27,857 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:27,857 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,857 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:27,857 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:27,869 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:27,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:27,870 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:27,870 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:27,870 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:27,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:27,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:27,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:27,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:27,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:27,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22bd55d3[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:27,876 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:27,876 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:27,878 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:27,878 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,878 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:27,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:27,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:27,878 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:27,878 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:27,878 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:27,879 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:27,879 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,879 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:27,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:27,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:27,879 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:27,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:27,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:27,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:27,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:27,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:27,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:27,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:27,888 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:27,888 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:27,888 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:27,888 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:27,889 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:27,894 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:27,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:27,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:27,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:27,894 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:27,895 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:27,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:27,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:27,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:27,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:27,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:27,895 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:27,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:27,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:27,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:27,896 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:27,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:27,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,898 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,898 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:27,898 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:27,898 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,898 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:27,898 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:27,898 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:27,898 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:27,899 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:27,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:27,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:27,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:27,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:27,899 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:27,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:27,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:27,900 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:27,900 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:27,923 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=304 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/A/0d8aa504581d4044b07483e2f5d04833 2023-03-20 10:58:27,939 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=304 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/B/b23380a9299748aaa5c6c69924359972 2023-03-20 10:58:27,950 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=304 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/C/7a6f551878f844a5b1260f726d2fd5d1 2023-03-20 10:58:27,954 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/A/0d8aa504581d4044b07483e2f5d04833 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/0d8aa504581d4044b07483e2f5d04833 2023-03-20 10:58:27,957 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:27,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52180 deadline: 1679309967946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:27,959 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/0d8aa504581d4044b07483e2f5d04833, entries=200, sequenceid=304, filesize=14.3 K 2023-03-20 10:58:27,961 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/B/b23380a9299748aaa5c6c69924359972 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/b23380a9299748aaa5c6c69924359972 2023-03-20 10:58:27,977 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/b23380a9299748aaa5c6c69924359972, entries=150, sequenceid=304, filesize=11.9 K 2023-03-20 10:58:27,979 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/C/7a6f551878f844a5b1260f726d2fd5d1 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/7a6f551878f844a5b1260f726d2fd5d1 2023-03-20 10:58:28,000 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:28,000 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/7a6f551878f844a5b1260f726d2fd5d1, entries=150, sequenceid=304, filesize=11.9 K 2023-03-20 10:58:28,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:28,002 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 00ef1cabe628ad083ccaa287ccbbfb01 in 512ms, sequenceid=304, compaction requested=true 2023-03-20 10:58:28,002 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 00ef1cabe628ad083ccaa287ccbbfb01: 2023-03-20 10:58:28,002 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 10:58:28,003 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:58:28,003 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 10:58:28,003 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:58:28,003 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:58:28,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:28,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:28,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:28,005 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:28,005 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:28,005 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:28,005 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:28,005 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:28,005 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:28,006 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:28,006 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:28,006 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:28,012 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51362 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 10:58:28,012 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48922 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 10:58:28,012 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1912): 00ef1cabe628ad083ccaa287ccbbfb01/A is initiating minor compaction (all files) 2023-03-20 10:58:28,012 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): 00ef1cabe628ad083ccaa287ccbbfb01/B is initiating minor compaction (all files) 2023-03-20 10:58:28,012 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 00ef1cabe628ad083ccaa287ccbbfb01/A in TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:28,012 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 00ef1cabe628ad083ccaa287ccbbfb01/B in TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:28,012 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/c205c353a77f421e95819ebc08952346, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/974a9baf2d114c9aa5d90428dd5c5b13, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/af065d914f91473a913141b8f5953e25, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/0d8aa504581d4044b07483e2f5d04833] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp, totalSize=50.2 K 2023-03-20 10:58:28,012 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/4356b31834ab4ea99e4c06f88b9c8a31, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/8f08871fecc5470097849a1397920f86, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/e2d1d7ce6a8f4af99314b1d8d1f4f6e2, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/b23380a9299748aaa5c6c69924359972] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp, totalSize=47.8 K 2023-03-20 10:58:28,014 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:28,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:28,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:28,014 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting c205c353a77f421e95819ebc08952346, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=242, earliestPutTs=1679309905235 2023-03-20 10:58:28,014 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 4356b31834ab4ea99e4c06f88b9c8a31, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=242, earliestPutTs=1679309905235 2023-03-20 10:58:28,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:28,015 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 974a9baf2d114c9aa5d90428dd5c5b13, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=253, earliestPutTs=1679309905857 2023-03-20 10:58:28,015 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:28,015 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting af065d914f91473a913141b8f5953e25, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=292, earliestPutTs=1679309906515 2023-03-20 10:58:28,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:28,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:28,015 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 8f08871fecc5470097849a1397920f86, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=253, earliestPutTs=1679309905857 2023-03-20 10:58:28,015 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:28,015 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 0d8aa504581d4044b07483e2f5d04833, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=304, earliestPutTs=1679309907142 2023-03-20 10:58:28,016 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting e2d1d7ce6a8f4af99314b1d8d1f4f6e2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=292, earliestPutTs=1679309906515 2023-03-20 10:58:28,017 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting b23380a9299748aaa5c6c69924359972, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=304, earliestPutTs=1679309907149 2023-03-20 10:58:28,023 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:28,023 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:28,026 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:28,026 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:28,026 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:28,026 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:28,026 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:28,028 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:28,028 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:28,028 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 00ef1cabe628ad083ccaa287ccbbfb01 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-20 10:58:28,029 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=A 2023-03-20 10:58:28,029 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:28,029 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=B 2023-03-20 10:58:28,029 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:28,029 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=C 2023-03-20 10:58:28,029 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:28,031 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] throttle.PressureAwareThroughputController(145): 00ef1cabe628ad083ccaa287ccbbfb01#A#compaction#436 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:58:28,043 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): 00ef1cabe628ad083ccaa287ccbbfb01#B#compaction#437 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:58:28,139 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/B/3a85d29595c24ac49b934b5d73e5edfb as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/3a85d29595c24ac49b934b5d73e5edfb 2023-03-20 10:58:28,149 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 00ef1cabe628ad083ccaa287ccbbfb01/B of 00ef1cabe628ad083ccaa287ccbbfb01 into 3a85d29595c24ac49b934b5d73e5edfb(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:58:28,149 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 00ef1cabe628ad083ccaa287ccbbfb01: 2023-03-20 10:58:28,149 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01., storeName=00ef1cabe628ad083ccaa287ccbbfb01/B, priority=12, startTime=1679309908003; duration=0sec 2023-03-20 10:58:28,150 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:58:28,150 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 10:58:28,163 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48888 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 10:58:28,163 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): 00ef1cabe628ad083ccaa287ccbbfb01/C is initiating minor compaction (all files) 2023-03-20 10:58:28,163 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 00ef1cabe628ad083ccaa287ccbbfb01/C in TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:28,163 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/ce7e68b49a854fb5848a591dae576208, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/c6617e014ddf4699a5629c6f2e59b088, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/e189b48dcee64e089821662663d83bc7, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/7a6f551878f844a5b1260f726d2fd5d1] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp, totalSize=47.7 K 2023-03-20 10:58:28,168 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting ce7e68b49a854fb5848a591dae576208, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=242, earliestPutTs=1679309905235 2023-03-20 10:58:28,170 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting c6617e014ddf4699a5629c6f2e59b088, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=253, earliestPutTs=1679309905857 2023-03-20 10:58:28,170 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting e189b48dcee64e089821662663d83bc7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=292, earliestPutTs=1679309906515 2023-03-20 10:58:28,171 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 7a6f551878f844a5b1260f726d2fd5d1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=304, earliestPutTs=1679309907149 2023-03-20 10:58:28,177 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): 00ef1cabe628ad083ccaa287ccbbfb01#C#compaction#439 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:58:28,195 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/C/1ab635d4435647eda5ef950d212c8dcf as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/1ab635d4435647eda5ef950d212c8dcf 2023-03-20 10:58:28,201 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 00ef1cabe628ad083ccaa287ccbbfb01/C of 00ef1cabe628ad083ccaa287ccbbfb01 into 1ab635d4435647eda5ef950d212c8dcf(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:58:28,201 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 00ef1cabe628ad083ccaa287ccbbfb01: 2023-03-20 10:58:28,201 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01., storeName=00ef1cabe628ad083ccaa287ccbbfb01/C, priority=12, startTime=1679309908003; duration=0sec 2023-03-20 10:58:28,201 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:58:28,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 00ef1cabe628ad083ccaa287ccbbfb01 2023-03-20 10:58:28,266 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. as already flushing 2023-03-20 10:58:28,334 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:28,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52180 deadline: 1679309968329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:28,441 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:28,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52180 deadline: 1679309968435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:28,512 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/A/1a8957c9c7d64a03ac40bcb322a0a01e as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/1a8957c9c7d64a03ac40bcb322a0a01e 2023-03-20 10:58:28,516 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 00ef1cabe628ad083ccaa287ccbbfb01/A of 00ef1cabe628ad083ccaa287ccbbfb01 into 1a8957c9c7d64a03ac40bcb322a0a01e(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:58:28,516 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 00ef1cabe628ad083ccaa287ccbbfb01: 2023-03-20 10:58:28,516 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01., storeName=00ef1cabe628ad083ccaa287ccbbfb01/A, priority=12, startTime=1679309908002; duration=0sec 2023-03-20 10:58:28,516 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:58:28,562 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=324 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/A/c4d9fa5e692c49029c913c67f37b0ce1 2023-03-20 10:58:28,598 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=324 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/B/e2ae5ec9a3a94bb086898d91885f375e 2023-03-20 10:58:28,607 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=324 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/C/744d16a4163f4b51b79fff199742c01b 2023-03-20 10:58:28,609 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/A/c4d9fa5e692c49029c913c67f37b0ce1 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/c4d9fa5e692c49029c913c67f37b0ce1 2023-03-20 10:58:28,612 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/c4d9fa5e692c49029c913c67f37b0ce1, entries=150, sequenceid=324, filesize=11.9 K 2023-03-20 10:58:28,612 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/B/e2ae5ec9a3a94bb086898d91885f375e as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/e2ae5ec9a3a94bb086898d91885f375e 2023-03-20 10:58:28,615 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/e2ae5ec9a3a94bb086898d91885f375e, entries=150, sequenceid=324, filesize=11.9 K 2023-03-20 10:58:28,616 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/C/744d16a4163f4b51b79fff199742c01b as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/744d16a4163f4b51b79fff199742c01b 2023-03-20 10:58:28,619 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/744d16a4163f4b51b79fff199742c01b, entries=150, sequenceid=324, filesize=11.9 K 2023-03-20 10:58:28,619 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 00ef1cabe628ad083ccaa287ccbbfb01 in 591ms, sequenceid=324, compaction requested=false 2023-03-20 10:58:28,619 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 00ef1cabe628ad083ccaa287ccbbfb01: 2023-03-20 10:58:28,620 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:28,620 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:28,620 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:28,620 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:28,620 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:28,620 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:28,621 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:28,621 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:28,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:28,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:28,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:28,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:28,621 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:28,621 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:28,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:28,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:28,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:28,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:28,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:28,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74f13959[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:28,622 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:28,622 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:28,623 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:28,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:28,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:28,623 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:28,623 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:28,623 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:28,623 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:28,623 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:28,624 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:28,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:28,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:28,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:28,624 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:28,624 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:28,624 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:28,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:28,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:28,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:28,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:28,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:28,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:28,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:28,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:28,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:28,626 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:28,626 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:28,626 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:28,626 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:28,626 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:28,626 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:28,626 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:28,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:28,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:28,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:28,627 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:28,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:28,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:28,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:28,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:28,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:28,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:28,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:28,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:28,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:28,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:28,628 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:28,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:28,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:28,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:28,638 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:28,639 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:28,639 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:28,639 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:28,639 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:28,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:28,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:28,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:28,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:28,639 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:28,640 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:28,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:28,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:28,640 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:28,640 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:28,640 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:28,640 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:28,640 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:28,640 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:28,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 00ef1cabe628ad083ccaa287ccbbfb01 2023-03-20 10:58:28,644 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 00ef1cabe628ad083ccaa287ccbbfb01 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-20 10:58:28,644 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=A 2023-03-20 10:58:28,644 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:28,644 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=B 2023-03-20 10:58:28,644 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:28,644 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=C 2023-03-20 10:58:28,644 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:28,650 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=340 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/A/efde1a1efeee43fb9926b2c0ad320083 2023-03-20 10:58:28,699 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:28,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52180 deadline: 1679309968692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:28,699 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:28,699 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:28,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52164 deadline: 1679309968693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:28,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52176 deadline: 1679309968694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:28,699 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:28,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52190 deadline: 1679309968694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:28,740 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:28,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:28,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:28,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:28,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:28,747 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:28,747 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:28,747 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:28,747 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:28,747 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:28,747 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:28,748 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:28,748 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:28,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:28,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:28,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:28,748 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:28,748 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:28,749 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:28,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:28,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:28,749 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:28,749 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:28,749 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:28,749 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:28,749 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:28,750 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:28,750 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:28,750 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:28,750 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:28,750 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. as already flushing 2023-03-20 10:58:28,750 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:28,750 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:28,750 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:28,750 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:28,750 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:28,750 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:28,751 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:28,751 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:28,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:28,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:28,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:28,752 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:28,752 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:28,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:28,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:28,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:28,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:28,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:28,753 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:28,753 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5edd034[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:28,753 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:28,753 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:28,753 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:28,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:28,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:28,754 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:28,754 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:28,754 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:28,754 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:28,754 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:28,754 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:28,754 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:28,754 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:28,755 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:28,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:28,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:28,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:28,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:28,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:28,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:28,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:28,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:28,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:28,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:28,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:28,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:28,758 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:28,758 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:28,758 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:28,758 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:28,758 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:28,759 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:28,759 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:28,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:28,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:28,759 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:28,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:28,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:28,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:28,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:28,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:28,759 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:28,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:28,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:28,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:28,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:28,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:28,760 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:28,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:28,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:28,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:28,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:28,763 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:28,763 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:28,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:28,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:28,764 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:28,763 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:28,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:28,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:28,764 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:28,764 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:28,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:28,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:28,764 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:28,764 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:28,765 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:28,765 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:28,765 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:28,765 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:28,765 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:28,807 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:28,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52164 deadline: 1679309968801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:28,807 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:28,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52180 deadline: 1679309968801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:28,808 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:28,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52176 deadline: 1679309968802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:28,808 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:28,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52190 deadline: 1679309968802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:28,864 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:28,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:28,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:28,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:28,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:28,886 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:28,886 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:28,886 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:28,886 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:28,888 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:28,888 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:28,889 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:28,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:28,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:28,889 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:28,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:28,890 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:28,890 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:28,890 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:28,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:28,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:28,892 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:28,906 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:28,906 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:28,906 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:28,906 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:28,906 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:28,906 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:28,906 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:28,907 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:28,907 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:28,907 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. as already flushing 2023-03-20 10:58:28,907 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:28,907 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:28,907 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:28,907 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:28,907 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:28,907 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:28,912 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:28,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:28,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:28,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:28,912 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:28,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:28,913 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:28,913 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:28,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:28,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:28,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:28,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:28,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:28,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55c4e0da[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:28,916 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:28,916 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:28,920 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:28,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:28,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:28,920 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:28,920 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:28,920 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:28,921 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:28,921 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:28,921 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:28,921 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:28,921 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:28,921 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:28,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:28,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:28,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:28,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:28,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:28,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:28,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:28,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:28,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:28,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:28,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:28,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:28,924 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:28,924 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:28,924 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:28,924 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:28,924 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:28,925 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:28,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:28,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:28,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:28,926 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:28,926 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:28,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:28,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:28,926 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:28,927 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:28,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:28,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:28,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:28,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:28,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:28,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:28,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:28,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:28,930 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:28,930 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:28,930 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:28,930 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:28,930 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:28,930 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:28,930 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:28,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:28,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:28,930 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:28,930 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:28,930 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:28,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:28,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:28,930 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:28,930 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:28,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:28,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:28,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:28,932 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:29,012 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:29,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52164 deadline: 1679309969008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,012 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:29,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52180 deadline: 1679309969008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,012 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:29,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52176 deadline: 1679309969009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,012 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:29,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52190 deadline: 1679309969010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,031 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:29,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:29,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:29,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:29,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:29,033 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:29,033 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:29,033 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:29,033 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:29,034 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:29,034 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:29,034 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:29,034 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:29,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:29,035 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,035 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:29,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:29,035 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:29,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:29,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:29,035 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:29,036 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:29,036 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:29,036 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:29,036 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:29,036 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:29,036 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:29,036 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:29,036 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:29,036 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:29,036 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. as already flushing 2023-03-20 10:58:29,036 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:29,036 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:29,036 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:29,036 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,036 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:29,036 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:29,037 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:29,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:29,037 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:29,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:29,038 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:29,038 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:29,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:29,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:29,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:29,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:29,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d1f347c[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:29,039 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:29,039 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:29,040 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:29,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:29,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:29,040 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:29,040 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:29,040 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:29,040 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,040 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:29,040 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,040 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:29,040 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:29,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:29,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:29,041 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:29,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:29,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:29,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:29,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:29,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:29,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:29,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:29,042 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:29,042 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:29,042 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:29,042 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:29,042 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:29,043 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:29,043 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:29,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:29,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:29,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:29,043 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:29,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:29,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:29,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:29,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:29,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:29,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:29,044 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:29,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:29,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:29,045 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:29,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:29,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,046 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:29,046 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:29,047 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:29,046 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:29,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:29,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:29,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:29,046 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:29,047 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:29,047 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,047 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:29,047 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:29,047 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:29,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:29,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:29,047 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:29,047 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:29,061 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=340 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/B/8c3417ca609347ec9f19eb82eafb405f 2023-03-20 10:58:29,147 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:29,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:29,149 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:29,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:29,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:29,150 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:29,150 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:29,150 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:29,150 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:29,150 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:29,150 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:29,151 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:29,151 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:29,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:29,151 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:29,151 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:29,152 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:29,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:29,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:29,152 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:29,152 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:29,152 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:29,152 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:29,152 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:29,152 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:29,152 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:29,152 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:29,153 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:29,153 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:29,153 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. as already flushing 2023-03-20 10:58:29,153 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:29,153 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:29,153 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:29,153 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,153 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:29,153 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:29,154 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:29,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:29,154 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:29,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:29,154 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:29,154 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:29,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:29,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:29,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:29,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:29,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6059ed77[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:29,155 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:29,155 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:29,156 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:29,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:29,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:29,156 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:29,156 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:29,156 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:29,156 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,156 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:29,156 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,156 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:29,156 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:29,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:29,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:29,157 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:29,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:29,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:29,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:29,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:29,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:29,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:29,158 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:29,158 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:29,158 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:29,158 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:29,158 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:29,158 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:29,159 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:29,159 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:29,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:29,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:29,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:29,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:29,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:29,159 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:29,160 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:29,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:29,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:29,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:29,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:29,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:29,160 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:29,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:29,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:29,161 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,161 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:29,161 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:29,162 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:29,161 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:29,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:29,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:29,162 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:29,162 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,162 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:29,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:29,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:29,162 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:29,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:29,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:29,162 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:29,162 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:29,162 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:29,162 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:29,262 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:29,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:29,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:29,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:29,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:29,265 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:29,265 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:29,265 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:29,265 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:29,265 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:29,265 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:29,267 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:29,267 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:29,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:29,267 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,267 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:29,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:29,268 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:29,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:29,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:29,268 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:29,268 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:29,268 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:29,268 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:29,268 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:29,268 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:29,268 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:29,269 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:29,270 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:29,270 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:29,270 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. as already flushing 2023-03-20 10:58:29,270 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:29,270 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:29,270 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:29,270 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,270 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:29,270 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:29,271 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,271 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:29,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:29,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:29,271 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:29,271 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:29,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:29,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:29,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:29,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:29,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:29,273 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@bdeed38[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:29,273 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:29,273 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:29,273 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:29,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:29,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:29,273 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:29,273 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:29,273 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:29,274 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,274 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:29,275 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:29,275 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:29,275 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:29,275 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:29,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:29,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:29,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:29,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:29,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:29,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:29,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:29,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:29,277 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:29,277 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:29,277 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:29,277 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:29,277 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:29,278 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:29,278 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:29,278 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:29,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:29,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:29,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:29,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:29,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:29,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:29,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:29,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:29,278 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:29,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:29,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:29,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:29,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,279 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:29,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:29,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,282 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:29,282 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:29,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:29,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:29,282 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,282 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:29,282 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:29,282 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,282 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:29,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:29,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:29,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:29,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:29,282 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:29,283 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:29,283 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:29,283 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:29,283 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:29,317 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:29,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52180 deadline: 1679309969314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,318 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:29,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52190 deadline: 1679309969314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,319 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:29,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52164 deadline: 1679309969314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,319 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:29,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52176 deadline: 1679309969315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,383 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:29,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:29,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:29,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:29,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:29,385 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:29,385 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:29,385 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:29,385 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:29,386 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:29,386 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:29,386 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,386 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:29,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:29,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:29,387 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,387 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:29,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:29,387 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:29,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:29,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:29,387 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:29,387 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:29,387 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:29,387 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:29,388 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:29,388 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:29,388 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:29,388 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:29,388 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:29,388 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:29,388 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. as already flushing 2023-03-20 10:58:29,388 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:29,388 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:29,388 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:29,388 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,388 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:29,388 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:29,389 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,389 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:29,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:29,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:29,389 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:29,389 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:29,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:29,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:29,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:29,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:29,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:29,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@796e1ea9[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:29,390 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:29,390 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:29,391 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:29,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:29,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:29,391 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:29,391 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,391 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:29,391 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:29,391 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:29,392 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:29,392 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,392 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:29,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:29,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:29,392 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:29,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:29,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:29,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:29,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:29,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:29,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:29,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:29,393 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:29,393 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:29,393 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:29,393 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:29,393 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:29,394 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:29,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:29,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:29,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:29,394 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:29,394 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:29,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:29,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:29,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:29,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:29,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:29,395 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:29,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:29,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:29,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:29,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,396 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:29,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:29,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,397 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,397 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:29,397 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:29,397 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:29,397 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:29,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:29,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:29,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:29,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:29,397 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:29,398 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:29,397 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:29,397 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:29,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:29,398 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:29,398 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:29,398 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:29,470 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=340 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/C/4ad090d16856438e9743f3d3df12656b 2023-03-20 10:58:29,477 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/A/efde1a1efeee43fb9926b2c0ad320083 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/efde1a1efeee43fb9926b2c0ad320083 2023-03-20 10:58:29,481 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/efde1a1efeee43fb9926b2c0ad320083, entries=200, sequenceid=340, filesize=14.3 K 2023-03-20 10:58:29,481 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/B/8c3417ca609347ec9f19eb82eafb405f as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/8c3417ca609347ec9f19eb82eafb405f 2023-03-20 10:58:29,489 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/8c3417ca609347ec9f19eb82eafb405f, entries=150, sequenceid=340, filesize=11.9 K 2023-03-20 10:58:29,490 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/C/4ad090d16856438e9743f3d3df12656b as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/4ad090d16856438e9743f3d3df12656b 2023-03-20 10:58:29,495 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/4ad090d16856438e9743f3d3df12656b, entries=150, sequenceid=340, filesize=11.9 K 2023-03-20 10:58:29,496 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 00ef1cabe628ad083ccaa287ccbbfb01 in 852ms, sequenceid=340, compaction requested=true 2023-03-20 10:58:29,496 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 00ef1cabe628ad083ccaa287ccbbfb01: 2023-03-20 10:58:29,496 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:58:29,496 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 10:58:29,496 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 10:58:29,496 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:58:29,496 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:58:29,497 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 10:58:29,497 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39612 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 10:58:29,497 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): 00ef1cabe628ad083ccaa287ccbbfb01/B is initiating minor compaction (all files) 2023-03-20 10:58:29,497 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1912): 00ef1cabe628ad083ccaa287ccbbfb01/A is initiating minor compaction (all files) 2023-03-20 10:58:29,497 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 00ef1cabe628ad083ccaa287ccbbfb01/B in TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:29,497 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 00ef1cabe628ad083ccaa287ccbbfb01/A in TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:29,497 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/3a85d29595c24ac49b934b5d73e5edfb, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/e2ae5ec9a3a94bb086898d91885f375e, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/8c3417ca609347ec9f19eb82eafb405f] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp, totalSize=36.3 K 2023-03-20 10:58:29,497 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/1a8957c9c7d64a03ac40bcb322a0a01e, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/c4d9fa5e692c49029c913c67f37b0ce1, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/efde1a1efeee43fb9926b2c0ad320083] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp, totalSize=38.7 K 2023-03-20 10:58:29,498 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 3a85d29595c24ac49b934b5d73e5edfb, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=304, earliestPutTs=1679309907149 2023-03-20 10:58:29,498 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 1a8957c9c7d64a03ac40bcb322a0a01e, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=304, earliestPutTs=1679309907149 2023-03-20 10:58:29,498 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:29,498 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting e2ae5ec9a3a94bb086898d91885f375e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=324, earliestPutTs=1679309907620 2023-03-20 10:58:29,498 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting c4d9fa5e692c49029c913c67f37b0ce1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=324, earliestPutTs=1679309907620 2023-03-20 10:58:29,498 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 8c3417ca609347ec9f19eb82eafb405f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=340, earliestPutTs=1679309908303 2023-03-20 10:58:29,498 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting efde1a1efeee43fb9926b2c0ad320083, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=340, earliestPutTs=1679309908303 2023-03-20 10:58:29,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:29,500 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:29,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:29,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:29,502 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:29,502 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:29,502 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:29,502 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:29,503 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:29,503 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:29,503 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): 00ef1cabe628ad083ccaa287ccbbfb01#B#compaction#445 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:58:29,503 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] throttle.PressureAwareThroughputController(145): 00ef1cabe628ad083ccaa287ccbbfb01#A#compaction#446 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:58:29,508 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:29,508 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:29,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:29,509 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,509 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:29,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:29,509 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:29,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:29,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:29,509 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:29,518 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:29,518 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:29,518 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:29,518 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:29,518 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:29,522 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:29,522 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:29,522 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:29,522 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:29,522 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 00ef1cabe628ad083ccaa287ccbbfb01 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-20 10:58:29,522 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=A 2023-03-20 10:58:29,522 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:29,523 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=B 2023-03-20 10:58:29,523 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:29,523 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=C 2023-03-20 10:58:29,523 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:29,540 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/B/50a474adf301439fb58dd9cb295ae24e as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/50a474adf301439fb58dd9cb295ae24e 2023-03-20 10:58:29,541 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/A/194e7bf2c6c54d53887b2a5242c308a0 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/194e7bf2c6c54d53887b2a5242c308a0 2023-03-20 10:58:29,544 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 00ef1cabe628ad083ccaa287ccbbfb01/A of 00ef1cabe628ad083ccaa287ccbbfb01 into 194e7bf2c6c54d53887b2a5242c308a0(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:58:29,544 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 00ef1cabe628ad083ccaa287ccbbfb01/B of 00ef1cabe628ad083ccaa287ccbbfb01 into 50a474adf301439fb58dd9cb295ae24e(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:58:29,544 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 00ef1cabe628ad083ccaa287ccbbfb01: 2023-03-20 10:58:29,544 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 00ef1cabe628ad083ccaa287ccbbfb01: 2023-03-20 10:58:29,544 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01., storeName=00ef1cabe628ad083ccaa287ccbbfb01/A, priority=13, startTime=1679309909496; duration=0sec 2023-03-20 10:58:29,544 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01., storeName=00ef1cabe628ad083ccaa287ccbbfb01/B, priority=13, startTime=1679309909496; duration=0sec 2023-03-20 10:58:29,544 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:58:29,544 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:58:29,544 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 10:58:29,545 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37138 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 10:58:29,545 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1912): 00ef1cabe628ad083ccaa287ccbbfb01/C is initiating minor compaction (all files) 2023-03-20 10:58:29,545 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 00ef1cabe628ad083ccaa287ccbbfb01/C in TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:29,545 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/1ab635d4435647eda5ef950d212c8dcf, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/744d16a4163f4b51b79fff199742c01b, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/4ad090d16856438e9743f3d3df12656b] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp, totalSize=36.3 K 2023-03-20 10:58:29,545 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 1ab635d4435647eda5ef950d212c8dcf, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=304, earliestPutTs=1679309907149 2023-03-20 10:58:29,545 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 744d16a4163f4b51b79fff199742c01b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=324, earliestPutTs=1679309907620 2023-03-20 10:58:29,545 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 4ad090d16856438e9743f3d3df12656b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=340, earliestPutTs=1679309908303 2023-03-20 10:58:29,548 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=359 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/A/00018e346427427693d9eb2b89b8ad17 2023-03-20 10:58:29,550 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] throttle.PressureAwareThroughputController(145): 00ef1cabe628ad083ccaa287ccbbfb01#C#compaction#448 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:58:29,555 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=359 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/B/4f098ddb87f2443696c67bb63dbcc827 2023-03-20 10:58:29,558 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/C/d60e7f3f796e41e1ab229e8c1f483a9d as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/d60e7f3f796e41e1ab229e8c1f483a9d 2023-03-20 10:58:29,561 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 00ef1cabe628ad083ccaa287ccbbfb01/C of 00ef1cabe628ad083ccaa287ccbbfb01 into d60e7f3f796e41e1ab229e8c1f483a9d(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:58:29,561 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 00ef1cabe628ad083ccaa287ccbbfb01: 2023-03-20 10:58:29,561 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01., storeName=00ef1cabe628ad083ccaa287ccbbfb01/C, priority=13, startTime=1679309909496; duration=0sec 2023-03-20 10:58:29,562 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:58:29,562 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=359 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/C/dcb2407086cf4407abb1c3c951e8f2c4 2023-03-20 10:58:29,565 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/A/00018e346427427693d9eb2b89b8ad17 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/00018e346427427693d9eb2b89b8ad17 2023-03-20 10:58:29,568 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/00018e346427427693d9eb2b89b8ad17, entries=150, sequenceid=359, filesize=11.9 K 2023-03-20 10:58:29,569 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/B/4f098ddb87f2443696c67bb63dbcc827 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/4f098ddb87f2443696c67bb63dbcc827 2023-03-20 10:58:29,573 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/4f098ddb87f2443696c67bb63dbcc827, entries=150, sequenceid=359, filesize=11.9 K 2023-03-20 10:58:29,574 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/C/dcb2407086cf4407abb1c3c951e8f2c4 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/dcb2407086cf4407abb1c3c951e8f2c4 2023-03-20 10:58:29,578 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/dcb2407086cf4407abb1c3c951e8f2c4, entries=150, sequenceid=359, filesize=11.9 K 2023-03-20 10:58:29,578 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=0 B/0 for 00ef1cabe628ad083ccaa287ccbbfb01 in 56ms, sequenceid=359, compaction requested=false 2023-03-20 10:58:29,578 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 00ef1cabe628ad083ccaa287ccbbfb01: 2023-03-20 10:58:29,578 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:29,578 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:29,579 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:29,579 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,579 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:29,579 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:29,581 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:29,582 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:29,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:29,582 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:29,582 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:29,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:29,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:29,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:29,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:29,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:29,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1db6792c[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:29,584 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:29,584 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:29,585 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:29,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:29,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:29,585 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,586 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:29,586 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:29,586 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:29,586 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:29,587 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,587 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:29,587 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:29,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:29,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:29,587 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:29,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:29,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:29,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:29,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:29,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:29,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:29,591 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:29,591 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:29,591 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:29,591 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:29,591 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:29,591 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:29,595 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:29,595 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:29,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:29,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:29,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:29,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:29,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:29,595 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:29,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:29,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:29,595 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:29,596 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:29,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:29,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:29,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:29,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,598 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:29,598 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:29,598 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,598 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:29,598 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:29,598 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:29,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:29,598 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:29,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:29,598 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:29,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:29,598 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:29,598 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:29,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:29,598 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:29,598 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:29,599 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:29,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:29,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:29,699 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:29,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:29,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:29,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:29,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:29,711 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:29,711 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:29,711 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:29,711 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:29,711 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:29,711 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:29,714 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:29,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:29,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:29,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:29,715 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:29,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:29,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:29,716 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:29,717 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:29,718 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:29,718 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:29,718 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:29,718 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:29,722 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,723 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,723 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:29,729 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:29,729 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:29,729 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:29,733 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:29,733 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 00ef1cabe628ad083ccaa287ccbbfb01: 2023-03-20 10:58:29,733 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:29,736 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:29,736 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:29,737 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,737 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:29,737 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:29,739 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:29,739 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:29,739 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:29,742 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:29,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:29,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:29,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:29,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:29,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:29,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:29,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32ab1db8[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:29,758 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:29,758 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:29,760 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:29,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:29,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:29,760 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:29,760 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:29,760 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:29,760 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,760 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:29,761 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:29,761 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:29,762 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:29,766 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:29,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:29,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:29,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:29,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:29,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:29,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:29,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:29,772 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:29,772 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:29,772 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:29,772 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:29,772 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:29,773 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:29,776 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:29,776 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:29,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:29,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:29,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:29,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:29,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:29,778 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:29,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:29,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:29,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:29,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:29,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:29,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:29,781 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:29,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:29,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:29,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,783 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,788 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:29,788 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:29,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:29,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:29,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:29,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:29,788 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:29,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:29,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:29,790 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,790 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:29,790 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:29,790 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,790 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:29,790 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:29,790 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:29,791 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:29,791 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:29,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 00ef1cabe628ad083ccaa287ccbbfb01 2023-03-20 10:58:29,838 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 00ef1cabe628ad083ccaa287ccbbfb01 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-20 10:58:29,838 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=A 2023-03-20 10:58:29,838 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:29,838 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=B 2023-03-20 10:58:29,838 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:29,838 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=C 2023-03-20 10:58:29,838 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:29,888 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1679309909859, max timestamp=1679309909885Num uniques 859; 2023-03-20 10:58:29,888 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-20 10:58:29,888 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1679309909859, max timestamp=1679309909885Num uniques 859; 2023-03-20 10:58:29,888 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-20 10:58:29,888 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1679309909859, max timestamp=1679309909885Num uniques 859; 2023-03-20 10:58:29,888 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-20 10:58:29,891 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:29,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:29,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:29,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:29,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:29,922 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:29,922 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:29,922 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:29,922 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:29,923 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:29,923 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:29,930 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:29,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:29,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:29,930 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:29,931 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:29,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:29,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:29,932 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:29,934 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:29,934 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:29,934 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,934 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:29,934 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:29,934 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:29,934 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:29,938 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:29,938 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:29,942 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:29,942 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:29,942 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. as already flushing 2023-03-20 10:58:29,942 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:29,942 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:29,942 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:29,942 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,943 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:29,943 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:29,944 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:29,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:29,946 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:29,947 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:29,947 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:29,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:29,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:29,956 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:29,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52190 deadline: 1679309969940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,957 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:29,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52180 deadline: 1679309969941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,958 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:29,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52176 deadline: 1679309969946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:29,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,959 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:29,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52164 deadline: 1679309969948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:29,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:29,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ee081f8[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:29,960 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:29,960 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:29,962 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:29,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:29,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:29,962 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,962 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:29,962 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:29,962 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:29,962 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:29,963 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:29,963 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:29,963 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:29,963 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:29,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:29,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:29,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:29,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:29,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:29,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:29,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:29,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:29,965 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:29,966 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:29,966 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:29,966 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:29,966 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:29,967 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:29,967 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:29,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:29,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:29,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:29,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:29,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:29,968 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:29,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:29,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:29,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:29,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:29,968 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:29,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:29,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:29,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,971 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:29,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:29,974 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,974 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:29,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:29,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:29,974 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:29,974 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:29,974 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:29,975 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:29,974 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:29,975 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:29,975 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:29,975 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:29,975 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:29,974 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:29,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:29,975 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:29,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:29,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:29,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:30,060 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:30,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52190 deadline: 1679309970058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,069 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:30,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52180 deadline: 1679309970059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,069 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:30,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52176 deadline: 1679309970059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,069 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:30,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52164 deadline: 1679309970060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,075 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:30,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:30,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:30,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:30,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:30,080 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:30,080 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:30,080 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:30,080 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:30,081 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:30,081 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:30,082 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:30,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:30,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:30,082 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:30,082 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:30,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:30,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:30,083 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:30,083 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,083 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:30,094 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:30,094 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:30,094 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:30,094 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:30,094 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:30,094 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:30,094 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:30,095 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:30,095 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:30,095 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. as already flushing 2023-03-20 10:58:30,095 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:30,095 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:30,095 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:30,095 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,095 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:30,095 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:30,096 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:30,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:30,096 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:30,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:30,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:30,097 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:30,097 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:30,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:30,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:30,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:30,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@477255db[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:30,098 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:30,098 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:30,099 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:30,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:30,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:30,099 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:30,099 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:30,099 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:30,099 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,099 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:30,100 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:30,100 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:30,100 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:30,100 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:30,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:30,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:30,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:30,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:30,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:30,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:30,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:30,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:30,120 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:30,120 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:30,120 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:30,120 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:30,120 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:30,121 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:30,121 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:30,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:30,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:30,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:30,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:30,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:30,122 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,123 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,124 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:30,124 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:30,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:30,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:30,124 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:30,124 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:30,124 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,124 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:30,124 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:30,124 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,125 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:30,125 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:30,125 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:30,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:30,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:30,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:30,125 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:30,125 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:30,125 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:30,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:30,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:30,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:30,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:30,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:30,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:30,142 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:30,225 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:30,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:30,227 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:30,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:30,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:30,230 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:30,230 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:30,230 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:30,230 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:30,231 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:30,231 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:30,232 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:30,232 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:30,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:30,232 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,232 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:30,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:30,232 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:30,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:30,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:30,233 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:30,233 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:30,233 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:30,233 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:30,233 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:30,233 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:30,234 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:30,234 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:30,234 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:30,234 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:30,234 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. as already flushing 2023-03-20 10:58:30,238 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:30,238 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:30,238 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:30,238 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,238 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:30,238 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:30,239 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,239 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:30,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:30,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:30,239 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:30,239 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:30,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:30,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:30,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:30,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:30,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:30,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69a32b00[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:30,240 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:30,240 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:30,241 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:30,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:30,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:30,241 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:30,241 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:30,241 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,241 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:30,241 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:30,243 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,243 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:30,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:30,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:30,244 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:30,244 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:30,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:30,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:30,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:30,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:30,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:30,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:30,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:30,245 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:30,245 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:30,245 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:30,245 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:30,245 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:30,246 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:30,246 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:30,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:30,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:30,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:30,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:30,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:30,246 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:30,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:30,247 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:30,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:30,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:30,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:30,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:30,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:30,248 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:30,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:30,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,249 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:30,249 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:30,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:30,249 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:30,249 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:30,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:30,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:30,249 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:30,249 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:30,249 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,249 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:30,249 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:30,249 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:30,249 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:30,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:30,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:30,250 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:30,250 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:30,269 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:30,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52190 deadline: 1679309970262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,275 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:30,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52180 deadline: 1679309970271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,275 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:30,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52176 deadline: 1679309970271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,275 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:30,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52164 deadline: 1679309970271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,291 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=373 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/A/8c7e0e64f9c44d2d8cfd34ecdb68107b 2023-03-20 10:58:30,334 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=373 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/B/e40f73d32d324ab8acdc8fef5849e93c 2023-03-20 10:58:30,350 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:30,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:30,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:30,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:30,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:30,366 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:30,366 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:30,366 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:30,366 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:30,367 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:30,367 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:30,368 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:30,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:30,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:30,368 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:30,368 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,368 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:30,368 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:30,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:30,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:30,369 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:30,390 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:30,390 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:30,393 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:30,393 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:30,393 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:30,403 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:30,403 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:30,406 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:30,406 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:30,406 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. as already flushing 2023-03-20 10:58:30,406 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:30,406 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:30,406 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:30,407 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,407 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:30,407 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:30,408 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:30,409 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:30,409 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:30,410 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:30,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:30,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:30,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:30,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:30,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:30,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:30,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38fbd51[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:30,412 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:30,412 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:30,419 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:30,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:30,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:30,420 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:30,420 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:30,420 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:30,420 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,420 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:30,421 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:30,421 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:30,421 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,421 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:30,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:30,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:30,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:30,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:30,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:30,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:30,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:30,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:30,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:30,425 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:30,426 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:30,426 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:30,426 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:30,426 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:30,427 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:30,427 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:30,427 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:30,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:30,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:30,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:30,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:30,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:30,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:30,428 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:30,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:30,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:30,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:30,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:30,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:30,429 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:30,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:30,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,433 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:30,433 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:30,433 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:30,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:30,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:30,433 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,433 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:30,433 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:30,433 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,433 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:30,433 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:30,433 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:30,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:30,434 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:30,433 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:30,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:30,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:30,435 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:30,435 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:30,535 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:30,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:30,537 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:30,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:30,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:30,542 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:30,542 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:30,542 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:30,542 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:30,546 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:30,546 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:30,547 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,547 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:30,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:30,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:30,547 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,547 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:30,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:30,548 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:30,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:30,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:30,548 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:30,550 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:30,550 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:30,550 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:30,550 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:30,550 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:30,558 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:30,558 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:30,558 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:30,558 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:30,558 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. as already flushing 2023-03-20 10:58:30,558 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:30,558 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:30,558 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:30,558 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,558 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:30,558 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:30,562 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:30,562 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:30,562 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:30,563 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:30,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:30,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:30,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:30,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:30,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:30,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:30,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49aa1399[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:30,566 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:30,566 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:30,569 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:30,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:30,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:30,570 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:30,570 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:30,570 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:30,570 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,570 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:30,571 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:30,571 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:30,571 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,571 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:30,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:30,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:30,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:30,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:30,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:30,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:30,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:30,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:30,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:30,575 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:30,575 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:30,575 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:30,575 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:30,575 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:30,577 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,577 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:30,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:30,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:30,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:30,577 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:30,577 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:30,577 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:30,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:30,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:30,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:30,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:30,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:30,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:30,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:30,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:30,579 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,579 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:30,579 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:30,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:30,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:30,579 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:30,579 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,580 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:30,579 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:30,580 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:30,580 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:30,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:30,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:30,580 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:30,580 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:30,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:30,580 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:30,580 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:30,580 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:30,580 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:30,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52190 deadline: 1679309970574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,581 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:30,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52180 deadline: 1679309970575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,581 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:30,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52164 deadline: 1679309970576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,581 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:30,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52176 deadline: 1679309970577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,582 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:30,680 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:30,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:30,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:30,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:30,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:30,685 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:30,685 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:30,685 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:30,685 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:30,689 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:30,689 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:30,690 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:30,690 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:30,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:30,691 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,691 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:30,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:30,691 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:30,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:30,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:30,691 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:30,694 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:30,694 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:30,694 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:30,694 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:30,694 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:30,694 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:30,694 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:30,694 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:30,694 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:30,694 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. as already flushing 2023-03-20 10:58:30,695 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:30,695 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:30,695 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:30,695 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,695 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:30,695 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:30,696 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,696 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:30,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:30,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:30,697 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:30,697 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:30,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:30,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:30,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:30,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:30,698 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:30,698 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c235bfa[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:30,698 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:30,698 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:30,698 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:30,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:30,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:30,698 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,699 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:30,699 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:30,699 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:30,699 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:30,699 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,699 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:30,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:30,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:30,699 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:30,700 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:30,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:30,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:30,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:30,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:30,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:30,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:30,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:30,701 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:30,701 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:30,701 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:30,701 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:30,701 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:30,702 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:30,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:30,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:30,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:30,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:30,703 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:30,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:30,704 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:30,704 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:30,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:30,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:30,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:30,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:30,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:30,705 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:30,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:30,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:30,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,721 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:30,721 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:30,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:30,721 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:30,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:30,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:30,721 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:30,721 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:30,721 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:30,721 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,721 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:30,721 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:30,721 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:30,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:30,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:30,722 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:30,722 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:30,722 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:30,807 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=373 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/C/b5429c5acff849419ef3f9d176c90022 2023-03-20 10:58:30,811 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/A/8c7e0e64f9c44d2d8cfd34ecdb68107b as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/8c7e0e64f9c44d2d8cfd34ecdb68107b 2023-03-20 10:58:30,819 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/8c7e0e64f9c44d2d8cfd34ecdb68107b, entries=250, sequenceid=373, filesize=16.7 K 2023-03-20 10:58:30,826 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:30,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:30,827 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/B/e40f73d32d324ab8acdc8fef5849e93c as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/e40f73d32d324ab8acdc8fef5849e93c 2023-03-20 10:58:30,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:30,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:30,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:30,830 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:30,830 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:30,830 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:30,830 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:30,830 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:30,830 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:30,831 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,832 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,832 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:30,833 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/e40f73d32d324ab8acdc8fef5849e93c, entries=150, sequenceid=373, filesize=11.9 K 2023-03-20 10:58:30,834 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/C/b5429c5acff849419ef3f9d176c90022 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/b5429c5acff849419ef3f9d176c90022 2023-03-20 10:58:30,834 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:30,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:30,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:30,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:30,835 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:30,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:30,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:30,836 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:30,836 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:30,836 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:30,836 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:30,836 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:30,836 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:30,836 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:30,836 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:30,836 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:30,837 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:30,837 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. as already flushing 2023-03-20 10:58:30,837 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:30,837 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:30,837 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:30,837 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,837 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:30,838 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:30,838 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/b5429c5acff849419ef3f9d176c90022, entries=150, sequenceid=373, filesize=11.9 K 2023-03-20 10:58:30,838 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 00ef1cabe628ad083ccaa287ccbbfb01 in 1000ms, sequenceid=373, compaction requested=true 2023-03-20 10:58:30,839 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 00ef1cabe628ad083ccaa287ccbbfb01: 2023-03-20 10:58:30,839 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:58:30,839 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 10:58:30,839 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:58:30,839 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 10:58:30,839 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:58:30,840 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42154 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 10:58:30,840 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 10:58:30,840 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1912): 00ef1cabe628ad083ccaa287ccbbfb01/A is initiating minor compaction (all files) 2023-03-20 10:58:30,840 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): 00ef1cabe628ad083ccaa287ccbbfb01/B is initiating minor compaction (all files) 2023-03-20 10:58:30,840 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 00ef1cabe628ad083ccaa287ccbbfb01/A in TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:30,840 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 00ef1cabe628ad083ccaa287ccbbfb01/B in TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:30,840 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/194e7bf2c6c54d53887b2a5242c308a0, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/00018e346427427693d9eb2b89b8ad17, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/8c7e0e64f9c44d2d8cfd34ecdb68107b] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp, totalSize=41.2 K 2023-03-20 10:58:30,840 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/50a474adf301439fb58dd9cb295ae24e, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/4f098ddb87f2443696c67bb63dbcc827, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/e40f73d32d324ab8acdc8fef5849e93c] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp, totalSize=36.4 K 2023-03-20 10:58:30,840 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 194e7bf2c6c54d53887b2a5242c308a0, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=340, earliestPutTs=1679309908303 2023-03-20 10:58:30,840 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 50a474adf301439fb58dd9cb295ae24e, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=340, earliestPutTs=1679309908303 2023-03-20 10:58:30,840 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 00018e346427427693d9eb2b89b8ad17, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=359, earliestPutTs=1679309908692 2023-03-20 10:58:30,841 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 4f098ddb87f2443696c67bb63dbcc827, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=359, earliestPutTs=1679309908692 2023-03-20 10:58:30,841 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 8c7e0e64f9c44d2d8cfd34ecdb68107b, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=373, earliestPutTs=1679309909821 2023-03-20 10:58:30,841 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting e40f73d32d324ab8acdc8fef5849e93c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=373, earliestPutTs=1679309909833 2023-03-20 10:58:30,845 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] throttle.PressureAwareThroughputController(145): 00ef1cabe628ad083ccaa287ccbbfb01#A#compaction#454 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:58:30,845 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): 00ef1cabe628ad083ccaa287ccbbfb01#B#compaction#455 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:58:30,855 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:30,855 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,855 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:30,856 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:30,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:30,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:30,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:30,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:30,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:30,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:30,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:30,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ca89855[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:30,892 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:30,892 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:30,896 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:30,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:30,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:30,896 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:30,896 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:30,896 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:30,896 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/B/6e4e66df9a394e28aa66f0ebb6c22a25 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/6e4e66df9a394e28aa66f0ebb6c22a25 2023-03-20 10:58:30,900 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:30,900 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:30,900 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:30,902 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 00ef1cabe628ad083ccaa287ccbbfb01/B of 00ef1cabe628ad083ccaa287ccbbfb01 into 6e4e66df9a394e28aa66f0ebb6c22a25(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:58:30,902 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 00ef1cabe628ad083ccaa287ccbbfb01: 2023-03-20 10:58:30,902 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01., storeName=00ef1cabe628ad083ccaa287ccbbfb01/B, priority=13, startTime=1679309910839; duration=0sec 2023-03-20 10:58:30,902 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:58:30,902 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 10:58:30,903 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37240 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 10:58:30,903 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): 00ef1cabe628ad083ccaa287ccbbfb01/C is initiating minor compaction (all files) 2023-03-20 10:58:30,903 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 00ef1cabe628ad083ccaa287ccbbfb01/C in TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:30,903 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/d60e7f3f796e41e1ab229e8c1f483a9d, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/dcb2407086cf4407abb1c3c951e8f2c4, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/b5429c5acff849419ef3f9d176c90022] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp, totalSize=36.4 K 2023-03-20 10:58:30,903 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting d60e7f3f796e41e1ab229e8c1f483a9d, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=340, earliestPutTs=1679309908303 2023-03-20 10:58:30,904 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting dcb2407086cf4407abb1c3c951e8f2c4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=359, earliestPutTs=1679309908692 2023-03-20 10:58:30,904 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting b5429c5acff849419ef3f9d176c90022, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=373, earliestPutTs=1679309909833 2023-03-20 10:58:30,906 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,906 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,906 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:30,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:30,906 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/A/5612ae12ced0485588c8494c33d0960b as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/5612ae12ced0485588c8494c33d0960b 2023-03-20 10:58:30,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:30,913 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 00ef1cabe628ad083ccaa287ccbbfb01/A of 00ef1cabe628ad083ccaa287ccbbfb01 into 5612ae12ced0485588c8494c33d0960b(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:58:30,913 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 00ef1cabe628ad083ccaa287ccbbfb01: 2023-03-20 10:58:30,913 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01., storeName=00ef1cabe628ad083ccaa287ccbbfb01/A, priority=13, startTime=1679309910839; duration=0sec 2023-03-20 10:58:30,913 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:58:30,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:30,917 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): 00ef1cabe628ad083ccaa287ccbbfb01#C#compaction#456 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:58:30,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:30,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:30,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:30,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:30,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:30,921 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:30,921 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:30,922 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:30,922 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:30,922 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:30,922 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:30,924 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:30,924 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:30,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:30,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:30,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:30,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:30,924 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:30,925 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,927 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:30,927 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:30,927 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:30,928 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:30,928 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:30,930 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:30,930 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:30,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:30,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:30,930 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:30,930 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:30,930 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:30,930 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:30,930 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:30,930 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:30,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:30,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:30,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:30,934 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:30,934 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:30,934 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:30,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:30,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:30,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:30,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:30,937 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/C/3035d4c7048d43d1805a20c6bec51161 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/3035d4c7048d43d1805a20c6bec51161 2023-03-20 10:58:30,951 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 00ef1cabe628ad083ccaa287ccbbfb01/C of 00ef1cabe628ad083ccaa287ccbbfb01 into 3035d4c7048d43d1805a20c6bec51161(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:58:30,951 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 00ef1cabe628ad083ccaa287ccbbfb01: 2023-03-20 10:58:30,951 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01., storeName=00ef1cabe628ad083ccaa287ccbbfb01/C, priority=13, startTime=1679309910839; duration=0sec 2023-03-20 10:58:30,951 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:58:31,028 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:31,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:31,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:31,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:31,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:31,038 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:31,038 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:31,038 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:31,038 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:31,040 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:31,040 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:31,049 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:31,049 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:31,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:31,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:31,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:31,050 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:31,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:31,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:31,051 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:31,051 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:31,051 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:31,051 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:31,051 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:31,051 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:31,051 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:31,052 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:31,052 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:31,052 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:31,052 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:31,052 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:31,052 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 00ef1cabe628ad083ccaa287ccbbfb01 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-03-20 10:58:31,052 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=A 2023-03-20 10:58:31,052 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:58:31,052 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=B 2023-03-20 10:58:31,052 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:58:31,052 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=C 2023-03-20 10:58:31,052 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:58:31,093 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. as already flushing 2023-03-20 10:58:31,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 00ef1cabe628ad083ccaa287ccbbfb01 2023-03-20 10:58:31,109 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:31,109 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:31,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52176 deadline: 1679309971095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:31,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52190 deadline: 1679309971095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:31,123 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:31,123 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:31,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52180 deadline: 1679309971109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:31,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52164 deadline: 1679309971109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:31,223 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:31,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52190 deadline: 1679309971223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:31,231 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:31,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52176 deadline: 1679309971226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:31,238 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:31,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52164 deadline: 1679309971234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:31,238 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:31,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52180 deadline: 1679309971234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:31,443 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:31,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52176 deadline: 1679309971436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:31,444 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:31,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52190 deadline: 1679309971438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:31,448 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:31,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52164 deadline: 1679309971447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:31,459 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:31,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52180 deadline: 1679309971449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:31,472 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=413 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/A/2a287692814448bd8632eb503a9b93fa 2023-03-20 10:58:31,522 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=413 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/B/3d73971d639447a3a78d7d6ebb77b884 2023-03-20 10:58:31,752 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:31,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52176 deadline: 1679309971744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:31,753 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:31,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52190 deadline: 1679309971746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:31,757 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:31,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52164 deadline: 1679309971752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:31,773 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:31,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52180 deadline: 1679309971767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:31,975 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=413 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/C/78a01a089e6249d29ef2c7dac3d2f164 2023-03-20 10:58:31,979 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/A/2a287692814448bd8632eb503a9b93fa as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/2a287692814448bd8632eb503a9b93fa 2023-03-20 10:58:31,984 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/2a287692814448bd8632eb503a9b93fa, entries=150, sequenceid=413, filesize=11.9 K 2023-03-20 10:58:31,984 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/B/3d73971d639447a3a78d7d6ebb77b884 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/3d73971d639447a3a78d7d6ebb77b884 2023-03-20 10:58:31,990 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/3d73971d639447a3a78d7d6ebb77b884, entries=150, sequenceid=413, filesize=11.9 K 2023-03-20 10:58:31,990 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-03-20 10:58:31,991 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/C/78a01a089e6249d29ef2c7dac3d2f164 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/78a01a089e6249d29ef2c7dac3d2f164 2023-03-20 10:58:31,996 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/78a01a089e6249d29ef2c7dac3d2f164, entries=150, sequenceid=413, filesize=11.9 K 2023-03-20 10:58:31,997 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for 00ef1cabe628ad083ccaa287ccbbfb01 in 945ms, sequenceid=413, compaction requested=false 2023-03-20 10:58:31,997 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 00ef1cabe628ad083ccaa287ccbbfb01: 2023-03-20 10:58:31,997 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:31,997 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:31,997 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:31,997 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:31,997 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:31,997 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:31,998 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:31,998 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:31,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:31,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:31,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:31,999 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:31,999 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:31,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:31,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:31,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:32,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:32,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:32,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:32,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12b3d5a4[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:32,002 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:32,002 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:32,003 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:32,003 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:32,003 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:32,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:32,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:32,003 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:32,004 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:32,004 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:32,005 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:32,005 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:32,005 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:32,005 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:32,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:32,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:32,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:32,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:32,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:32,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:32,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:32,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:32,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:32,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:32,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:32,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:32,008 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:32,008 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:32,008 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:32,008 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:32,008 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:32,009 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:32,009 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:32,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:32,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:32,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:32,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:32,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:32,010 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:32,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:32,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:32,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:32,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:32,011 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:32,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:32,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:32,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:32,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:32,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:32,012 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:32,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:32,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:32,027 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:32,027 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:32,027 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:32,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:32,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:32,027 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:32,028 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:32,028 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:32,028 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:32,028 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:32,028 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:32,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:32,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:32,028 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:32,028 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:32,028 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:32,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:32,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:32,132 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:32,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:32,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:32,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:32,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:32,146 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:32,146 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:32,146 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:32,146 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:32,146 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:32,146 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:32,147 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:32,147 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:32,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:32,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:32,148 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:32,148 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:32,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:32,148 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:32,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:32,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:32,149 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:32,152 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:32,152 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:32,153 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:32,153 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:32,153 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:32,154 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:32,154 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:32,154 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:32,154 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:32,159 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 00ef1cabe628ad083ccaa287ccbbfb01 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-03-20 10:58:32,159 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=A 2023-03-20 10:58:32,159 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:32,159 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=B 2023-03-20 10:58:32,159 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:32,159 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=C 2023-03-20 10:58:32,159 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:32,195 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=422 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/A/ef4d97a8f7cc4832b30bf2fce5e93b05 2023-03-20 10:58:32,211 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=422 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/B/d077d587b1004f5bbfa31131d33d67a0 2023-03-20 10:58:32,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 00ef1cabe628ad083ccaa287ccbbfb01 2023-03-20 10:58:32,258 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. as already flushing 2023-03-20 10:58:32,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679309912293, max timestamp=1679309912300Num uniques 859; 2023-03-20 10:58:32,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-20 10:58:32,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679309912293, max timestamp=1679309912300Num uniques 859; 2023-03-20 10:58:32,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-20 10:58:32,304 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679309912293, max timestamp=1679309912300Num uniques 859; 2023-03-20 10:58:32,304 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-20 10:58:32,343 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:32,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52164 deadline: 1679309972337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:32,343 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:32,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52180 deadline: 1679309972338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:32,344 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:32,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52176 deadline: 1679309972338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:32,344 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:32,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52190 deadline: 1679309972338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:32,453 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:32,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52164 deadline: 1679309972448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:32,453 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:32,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52180 deadline: 1679309972448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:32,455 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:32,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52190 deadline: 1679309972450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:32,455 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:32,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52176 deadline: 1679309972450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:32,625 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=422 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/C/c41e0c6f991343e4963598edea8cf3d8 2023-03-20 10:58:32,629 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/A/ef4d97a8f7cc4832b30bf2fce5e93b05 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/ef4d97a8f7cc4832b30bf2fce5e93b05 2023-03-20 10:58:32,638 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/ef4d97a8f7cc4832b30bf2fce5e93b05, entries=150, sequenceid=422, filesize=11.9 K 2023-03-20 10:58:32,640 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/B/d077d587b1004f5bbfa31131d33d67a0 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/d077d587b1004f5bbfa31131d33d67a0 2023-03-20 10:58:32,649 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/d077d587b1004f5bbfa31131d33d67a0, entries=150, sequenceid=422, filesize=11.9 K 2023-03-20 10:58:32,651 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/C/c41e0c6f991343e4963598edea8cf3d8 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/c41e0c6f991343e4963598edea8cf3d8 2023-03-20 10:58:32,654 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/c41e0c6f991343e4963598edea8cf3d8, entries=150, sequenceid=422, filesize=11.9 K 2023-03-20 10:58:32,656 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 00ef1cabe628ad083ccaa287ccbbfb01 in 497ms, sequenceid=422, compaction requested=true 2023-03-20 10:58:32,656 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 00ef1cabe628ad083ccaa287ccbbfb01: 2023-03-20 10:58:32,656 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:32,656 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:32,657 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:32,657 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:32,657 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:32,657 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:32,661 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:32,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:32,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:32,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:32,662 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:32,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:32,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:32,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:32,664 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:32,664 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:32,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:32,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:32,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:32,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b1c2ab0[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:32,664 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:32,665 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:32,668 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679309912336, max timestamp=1679309912660Num uniques 1068; 2023-03-20 10:58:32,668 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679309912336, max timestamp=1679309912660Num uniques 1068; 2023-03-20 10:58:32,668 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-03-20 10:58:32,668 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-03-20 10:58:32,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 00ef1cabe628ad083ccaa287ccbbfb01 2023-03-20 10:58:32,668 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 00ef1cabe628ad083ccaa287ccbbfb01 3/3 column families, dataSize=67.09 KB heapSize=183.88 KB 2023-03-20 10:58:32,669 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=A 2023-03-20 10:58:32,669 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:58:32,669 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=B 2023-03-20 10:58:32,669 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:58:32,669 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=C 2023-03-20 10:58:32,669 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-20 10:58:32,669 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:32,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:32,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:32,670 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:32,670 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:32,670 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:32,670 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:32,670 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:32,671 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:32,671 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:32,671 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:32,671 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:32,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:32,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:32,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:32,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:32,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:32,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:32,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:32,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:32,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:32,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:32,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:32,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:32,676 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:32,676 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:32,676 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:32,676 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:32,676 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:32,688 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:32,688 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:32,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:32,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:32,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:32,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:32,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:32,689 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:32,690 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:32,690 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:32,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:32,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:32,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:32,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:32,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:32,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:32,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:32,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:32,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:32,704 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:32,704 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:32,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:32,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:32,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:32,704 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:32,704 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:32,704 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:32,704 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:32,704 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:32,704 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:32,705 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:32,705 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:32,705 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:32,705 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:32,704 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:32,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:32,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:32,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:32,715 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=461 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/A/d13857e2b8e24461a2ac292d89aa7bc2 2023-03-20 10:58:32,716 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:32,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52180 deadline: 1679309972702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:32,716 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:32,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52176 deadline: 1679309972703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:32,716 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:32,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52164 deadline: 1679309972705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:32,722 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:32,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52190 deadline: 1679309972707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:32,741 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=461 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/B/b3b3b8fdeae24f778c5d4d8559cbf279 2023-03-20 10:58:32,806 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:32,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:32,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:32,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:32,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:32,811 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:32,811 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:32,811 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:32,811 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:32,811 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:32,811 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:32,812 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:32,812 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:32,813 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:32,813 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:32,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:32,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:32,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:32,814 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:32,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:32,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:32,814 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:32,814 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:32,815 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:32,815 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:32,815 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:32,815 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:32,815 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:32,815 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:32,815 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:32,815 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:32,815 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. as already flushing 2023-03-20 10:58:32,815 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:32,815 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:32,815 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:32,815 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:32,815 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:32,815 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:32,817 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:32,817 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:32,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:32,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:32,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:32,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:32,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:32,818 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:32,818 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:32,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:32,818 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:32,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52180 deadline: 1679309972818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:32,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:32,819 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:32,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52164 deadline: 1679309972818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:32,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:32,819 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:32,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52176 deadline: 1679309972818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:32,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:32,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d665ca2[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:32,819 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:32,819 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:32,820 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:32,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:32,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:32,820 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:32,820 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:32,820 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:32,820 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:32,820 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:32,821 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:32,821 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:32,821 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:32,821 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:32,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:32,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:32,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:32,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:32,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:32,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:32,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:32,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:32,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:32,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:32,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:32,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:32,824 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:32,824 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:32,824 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:32,824 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:32,824 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:32,828 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:32,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52190 deadline: 1679309972823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:32,836 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:32,836 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:32,836 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:32,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:32,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:32,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:32,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:32,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:32,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:32,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:32,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:32,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:32,837 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:32,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:32,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:32,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:32,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:32,838 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:32,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:32,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:32,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:32,844 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:32,845 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:32,845 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:32,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:32,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:32,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:32,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:32,845 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:32,845 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:32,845 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:32,845 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:32,845 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:32,845 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:32,845 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:32,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:32,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:32,846 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:32,846 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:32,948 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:32,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:32,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:32,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:32,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:32,951 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:32,951 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:32,951 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:32,951 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:32,962 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:32,962 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:32,966 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:32,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:32,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:32,967 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:32,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:32,968 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:32,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:32,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:32,968 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:32,968 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:32,968 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:32,969 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:32,969 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:32,969 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:32,969 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:32,969 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:32,969 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:32,969 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:32,969 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. as already flushing 2023-03-20 10:58:32,969 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:32,969 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:32,969 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:32,969 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:32,969 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:32,969 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:32,970 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:32,970 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:32,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:32,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:32,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:32,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:32,971 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:32,971 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:32,970 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:32,971 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:32,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:32,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:32,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:32,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:32,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:32,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25a4176b[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:32,977 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:32,977 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:32,982 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:32,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:32,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:32,982 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:32,982 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:32,982 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:32,986 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:32,986 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:32,986 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:32,986 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:32,986 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:32,990 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:32,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:32,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:32,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:32,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:32,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:32,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:32,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:32,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:32,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:32,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:32,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:32,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:32,999 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:32,999 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:33,000 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:33,000 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:33,000 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:33,003 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:33,003 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:33,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:33,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:33,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:33,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:33,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:33,004 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:33,005 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:33,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:33,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:33,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:33,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:33,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:33,007 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:33,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:33,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:33,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:33,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:33,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:33,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:33,013 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:33,013 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:33,013 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:33,013 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:33,013 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:33,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:33,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:33,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:33,013 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:33,014 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:33,014 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:33,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:33,013 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:33,013 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:33,014 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:33,014 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:33,014 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:33,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:33,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:33,022 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:33,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52180 deadline: 1679309973021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:33,022 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:33,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52176 deadline: 1679309973021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:33,030 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:33,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52164 deadline: 1679309973027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:33,031 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:33,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52190 deadline: 1679309973029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:33,114 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:33,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:33,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:33,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:33,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:33,130 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:33,130 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:33,130 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:33,130 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:33,131 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:33,131 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:33,131 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:33,131 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:33,132 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:33,132 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:33,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:33,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:33,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:33,135 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:33,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:33,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:33,135 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:33,146 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:33,146 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:33,146 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:33,146 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:33,146 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:33,147 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:33,147 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:33,147 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:33,147 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:33,147 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. as already flushing 2023-03-20 10:58:33,148 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:33,148 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:33,148 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:33,148 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:33,148 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:33,148 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:33,150 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:33,151 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:33,151 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:33,151 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:33,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:33,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:33,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:33,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:33,152 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=461 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/C/2a1568ef043749d68091bf55f254c32e 2023-03-20 10:58:33,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:33,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:33,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:33,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:33,153 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:33,153 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29b6f88e[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:33,153 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:33,154 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:33,154 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:33,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:33,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:33,154 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:33,154 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:33,154 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:33,155 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:33,155 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:33,155 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:33,156 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:33,156 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:33,156 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:33,156 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/A/d13857e2b8e24461a2ac292d89aa7bc2 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/d13857e2b8e24461a2ac292d89aa7bc2 2023-03-20 10:58:33,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:33,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:33,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:33,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:33,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:33,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:33,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:33,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:33,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:33,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:33,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:33,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:33,159 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:33,159 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:33,159 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:33,159 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:33,159 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:33,159 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/d13857e2b8e24461a2ac292d89aa7bc2, entries=150, sequenceid=461, filesize=11.9 K 2023-03-20 10:58:33,160 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/B/b3b3b8fdeae24f778c5d4d8559cbf279 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/b3b3b8fdeae24f778c5d4d8559cbf279 2023-03-20 10:58:33,160 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:33,160 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:33,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:33,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:33,160 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:33,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:33,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:33,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:33,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:33,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:33,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:33,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:33,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:33,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:33,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:33,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:33,162 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:33,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:33,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:33,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:33,163 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/b3b3b8fdeae24f778c5d4d8559cbf279, entries=150, sequenceid=461, filesize=11.9 K 2023-03-20 10:58:33,164 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/C/2a1568ef043749d68091bf55f254c32e as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/2a1568ef043749d68091bf55f254c32e 2023-03-20 10:58:33,169 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:33,178 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:33,178 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:33,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:33,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:33,178 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:33,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:33,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:33,178 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/2a1568ef043749d68091bf55f254c32e, entries=150, sequenceid=461, filesize=11.9 K 2023-03-20 10:58:33,179 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:33,179 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:33,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:33,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:33,179 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:33,179 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:33,179 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:33,179 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:33,179 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:33,179 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:33,179 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:33,182 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~203.75 KB/208640, currentSize=107.34 KB/109920 for 00ef1cabe628ad083ccaa287ccbbfb01 in 514ms, sequenceid=461, compaction requested=true 2023-03-20 10:58:33,182 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 00ef1cabe628ad083ccaa287ccbbfb01: 2023-03-20 10:58:33,182 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:58:33,183 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 10:58:33,183 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 10:58:33,183 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:58:33,183 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:58:33,187 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49562 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 10:58:33,187 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49562 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 10:58:33,187 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): 00ef1cabe628ad083ccaa287ccbbfb01/B is initiating minor compaction (all files) 2023-03-20 10:58:33,187 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1912): 00ef1cabe628ad083ccaa287ccbbfb01/A is initiating minor compaction (all files) 2023-03-20 10:58:33,187 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 00ef1cabe628ad083ccaa287ccbbfb01/B in TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:33,187 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 00ef1cabe628ad083ccaa287ccbbfb01/A in TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:33,187 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/6e4e66df9a394e28aa66f0ebb6c22a25, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/3d73971d639447a3a78d7d6ebb77b884, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/d077d587b1004f5bbfa31131d33d67a0, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/b3b3b8fdeae24f778c5d4d8559cbf279] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp, totalSize=48.4 K 2023-03-20 10:58:33,187 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/5612ae12ced0485588c8494c33d0960b, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/2a287692814448bd8632eb503a9b93fa, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/ef4d97a8f7cc4832b30bf2fce5e93b05, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/d13857e2b8e24461a2ac292d89aa7bc2] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp, totalSize=48.4 K 2023-03-20 10:58:33,188 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 6e4e66df9a394e28aa66f0ebb6c22a25, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=373, earliestPutTs=1679309909833 2023-03-20 10:58:33,188 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 5612ae12ced0485588c8494c33d0960b, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=373, earliestPutTs=1679309909833 2023-03-20 10:58:33,188 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 3d73971d639447a3a78d7d6ebb77b884, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=413, earliestPutTs=1679309909928 2023-03-20 10:58:33,188 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting d077d587b1004f5bbfa31131d33d67a0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=422, earliestPutTs=1679309911094 2023-03-20 10:58:33,188 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 2a287692814448bd8632eb503a9b93fa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=413, earliestPutTs=1679309909928 2023-03-20 10:58:33,189 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting ef4d97a8f7cc4832b30bf2fce5e93b05, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=422, earliestPutTs=1679309911094 2023-03-20 10:58:33,189 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting b3b3b8fdeae24f778c5d4d8559cbf279, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=461, earliestPutTs=1679309912337 2023-03-20 10:58:33,189 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting d13857e2b8e24461a2ac292d89aa7bc2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=461, earliestPutTs=1679309912337 2023-03-20 10:58:33,205 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): 00ef1cabe628ad083ccaa287ccbbfb01#B#compaction#466 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:58:33,211 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] throttle.PressureAwareThroughputController(145): 00ef1cabe628ad083ccaa287ccbbfb01#A#compaction#467 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:58:33,255 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/B/caa40d24be7e40118a1e736f96a63e09 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/caa40d24be7e40118a1e736f96a63e09 2023-03-20 10:58:33,262 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 00ef1cabe628ad083ccaa287ccbbfb01/B of 00ef1cabe628ad083ccaa287ccbbfb01 into caa40d24be7e40118a1e736f96a63e09(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:58:33,262 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 00ef1cabe628ad083ccaa287ccbbfb01: 2023-03-20 10:58:33,262 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01., storeName=00ef1cabe628ad083ccaa287ccbbfb01/B, priority=12, startTime=1679309913183; duration=0sec 2023-03-20 10:58:33,262 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:58:33,262 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 10:58:33,263 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49528 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 10:58:33,263 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): 00ef1cabe628ad083ccaa287ccbbfb01/C is initiating minor compaction (all files) 2023-03-20 10:58:33,263 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 00ef1cabe628ad083ccaa287ccbbfb01/C in TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:33,264 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/3035d4c7048d43d1805a20c6bec51161, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/78a01a089e6249d29ef2c7dac3d2f164, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/c41e0c6f991343e4963598edea8cf3d8, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/2a1568ef043749d68091bf55f254c32e] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp, totalSize=48.4 K 2023-03-20 10:58:33,264 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 3035d4c7048d43d1805a20c6bec51161, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=373, earliestPutTs=1679309909833 2023-03-20 10:58:33,264 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 78a01a089e6249d29ef2c7dac3d2f164, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=413, earliestPutTs=1679309909928 2023-03-20 10:58:33,264 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting c41e0c6f991343e4963598edea8cf3d8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=422, earliestPutTs=1679309911094 2023-03-20 10:58:33,265 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 2a1568ef043749d68091bf55f254c32e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=461, earliestPutTs=1679309912337 2023-03-20 10:58:33,275 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): 00ef1cabe628ad083ccaa287ccbbfb01#C#compaction#468 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:58:33,279 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:33,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:33,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:33,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:33,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:33,294 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:33,294 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:33,294 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:33,294 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:33,294 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:33,295 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:33,297 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:33,297 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:33,297 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:33,298 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:33,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:33,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:33,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:33,301 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:33,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:33,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:33,301 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:33,301 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:33,301 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:33,301 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:33,301 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:33,301 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:33,302 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:33,302 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:33,302 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:33,302 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:33,302 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 00ef1cabe628ad083ccaa287ccbbfb01 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-20 10:58:33,302 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=A 2023-03-20 10:58:33,302 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:33,302 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=B 2023-03-20 10:58:33,302 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:33,302 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=C 2023-03-20 10:58:33,302 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:33,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 00ef1cabe628ad083ccaa287ccbbfb01 2023-03-20 10:58:33,329 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. as already flushing 2023-03-20 10:58:33,340 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:33,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52164 deadline: 1679309973335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:33,341 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:33,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52176 deadline: 1679309973336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:33,342 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:33,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52180 deadline: 1679309973339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:33,352 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:33,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52190 deadline: 1679309973341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:33,447 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:33,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52180 deadline: 1679309973446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:33,447 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:33,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52164 deadline: 1679309973446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:33,453 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:33,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52176 deadline: 1679309973450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:33,454 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:33,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52190 deadline: 1679309973454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:33,655 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:33,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52180 deadline: 1679309973649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:33,655 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:33,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52164 deadline: 1679309973649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:33,659 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:33,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52176 deadline: 1679309973654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:33,659 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:33,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52190 deadline: 1679309973655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:33,667 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/A/09f1d94a719242d6a6ec232a13252f10 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/09f1d94a719242d6a6ec232a13252f10 2023-03-20 10:58:33,671 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 00ef1cabe628ad083ccaa287ccbbfb01/A of 00ef1cabe628ad083ccaa287ccbbfb01 into 09f1d94a719242d6a6ec232a13252f10(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:58:33,671 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 00ef1cabe628ad083ccaa287ccbbfb01: 2023-03-20 10:58:33,671 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01., storeName=00ef1cabe628ad083ccaa287ccbbfb01/A, priority=12, startTime=1679309913182; duration=0sec 2023-03-20 10:58:33,671 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:58:33,731 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/C/cc2521458e4c45db817d9e61eabd2d0e as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/cc2521458e4c45db817d9e61eabd2d0e 2023-03-20 10:58:33,746 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 00ef1cabe628ad083ccaa287ccbbfb01/C of 00ef1cabe628ad083ccaa287ccbbfb01 into cc2521458e4c45db817d9e61eabd2d0e(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:58:33,746 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 00ef1cabe628ad083ccaa287ccbbfb01: 2023-03-20 10:58:33,746 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01., storeName=00ef1cabe628ad083ccaa287ccbbfb01/C, priority=12, startTime=1679309913183; duration=0sec 2023-03-20 10:58:33,747 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:58:33,750 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=481 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/A/0a900540558c4a53a06cd7238db3fb57 2023-03-20 10:58:33,760 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=481 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/B/3983ea79ff1c4636818fce7330fccbfd 2023-03-20 10:58:33,767 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=481 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/C/665612b9488a4c8f901ec084028c3314 2023-03-20 10:58:33,772 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/A/0a900540558c4a53a06cd7238db3fb57 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/0a900540558c4a53a06cd7238db3fb57 2023-03-20 10:58:33,777 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/0a900540558c4a53a06cd7238db3fb57, entries=150, sequenceid=481, filesize=11.9 K 2023-03-20 10:58:33,778 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/B/3983ea79ff1c4636818fce7330fccbfd as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/3983ea79ff1c4636818fce7330fccbfd 2023-03-20 10:58:33,781 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/3983ea79ff1c4636818fce7330fccbfd, entries=150, sequenceid=481, filesize=11.9 K 2023-03-20 10:58:33,782 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/C/665612b9488a4c8f901ec084028c3314 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/665612b9488a4c8f901ec084028c3314 2023-03-20 10:58:33,784 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/665612b9488a4c8f901ec084028c3314, entries=150, sequenceid=481, filesize=11.9 K 2023-03-20 10:58:33,785 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 00ef1cabe628ad083ccaa287ccbbfb01 in 483ms, sequenceid=481, compaction requested=false 2023-03-20 10:58:33,785 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 00ef1cabe628ad083ccaa287ccbbfb01: 2023-03-20 10:58:33,785 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:33,785 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:33,785 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:33,785 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:33,785 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:33,786 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:33,786 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:33,787 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:33,787 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:33,787 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:33,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:33,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:33,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:33,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:33,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:33,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:33,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:33,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:33,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:33,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43151ea2[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:33,791 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:33,791 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:33,792 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:33,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:33,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:33,792 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:33,792 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:33,792 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:33,792 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:33,792 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:33,794 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:33,794 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:33,795 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:33,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:33,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:33,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:33,795 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:33,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:33,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:33,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:33,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:33,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:33,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:33,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:33,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:33,796 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:33,796 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:33,796 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:33,796 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:33,796 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:33,796 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:33,797 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:33,797 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:33,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:33,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:33,798 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:33,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:33,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:33,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:33,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:33,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:33,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:33,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:33,800 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:33,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:33,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:33,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:33,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:33,803 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:33,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:33,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:33,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:33,811 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:33,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:33,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:33,812 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:33,812 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:33,812 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:33,812 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:33,812 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:33,812 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:33,812 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:33,812 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:33,812 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:33,814 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:33,814 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:33,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:33,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:33,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:33,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:33,912 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:33,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:33,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:33,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:33,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:33,922 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:33,922 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:33,922 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:33,922 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:33,930 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:33,930 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:33,931 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:33,931 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:33,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:33,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:33,932 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:33,932 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:33,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:33,932 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:33,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:33,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:33,933 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:33,938 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:33,938 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:33,938 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:33,938 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:33,938 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:33,946 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:33,946 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:33,946 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:33,946 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:33,946 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 00ef1cabe628ad083ccaa287ccbbfb01 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-20 10:58:33,946 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=A 2023-03-20 10:58:33,946 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:33,947 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=B 2023-03-20 10:58:33,947 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:33,947 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=C 2023-03-20 10:58:33,947 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:33,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 00ef1cabe628ad083ccaa287ccbbfb01 2023-03-20 10:58:33,960 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. as already flushing 2023-03-20 10:58:34,027 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:34,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52180 deadline: 1679309974016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:34,028 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:34,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52164 deadline: 1679309974019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:34,028 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:34,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52176 deadline: 1679309974020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:34,034 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:34,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52190 deadline: 1679309974025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:34,133 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:34,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52180 deadline: 1679309974129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:34,134 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:34,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52164 deadline: 1679309974129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:34,134 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:34,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52176 deadline: 1679309974130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:34,147 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:34,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52190 deadline: 1679309974136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:34,341 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:34,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52164 deadline: 1679309974335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:34,342 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:34,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52180 deadline: 1679309974336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:34,342 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:34,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52176 deadline: 1679309974336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:34,351 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:34,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52190 deadline: 1679309974349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:34,382 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=496 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/A/56b61faa6bec4e9c8c49a2794744e0dc 2023-03-20 10:58:34,423 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=496 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/B/18861f4e90f8416a801e645786b8fab0 2023-03-20 10:58:34,453 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=496 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/C/b292772694454a79a0d284bec3be81d4 2023-03-20 10:58:34,459 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/A/56b61faa6bec4e9c8c49a2794744e0dc as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/56b61faa6bec4e9c8c49a2794744e0dc 2023-03-20 10:58:34,468 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/56b61faa6bec4e9c8c49a2794744e0dc, entries=150, sequenceid=496, filesize=11.9 K 2023-03-20 10:58:34,469 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/B/18861f4e90f8416a801e645786b8fab0 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/18861f4e90f8416a801e645786b8fab0 2023-03-20 10:58:34,473 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/18861f4e90f8416a801e645786b8fab0, entries=150, sequenceid=496, filesize=11.9 K 2023-03-20 10:58:34,473 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/C/b292772694454a79a0d284bec3be81d4 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/b292772694454a79a0d284bec3be81d4 2023-03-20 10:58:34,484 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/b292772694454a79a0d284bec3be81d4, entries=150, sequenceid=496, filesize=11.9 K 2023-03-20 10:58:34,486 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 00ef1cabe628ad083ccaa287ccbbfb01 in 540ms, sequenceid=496, compaction requested=true 2023-03-20 10:58:34,486 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 00ef1cabe628ad083ccaa287ccbbfb01: 2023-03-20 10:58:34,486 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:34,486 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:34,486 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:34,486 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:34,486 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:34,486 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:34,491 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:34,491 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:34,491 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:34,492 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:34,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:34,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:34,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:34,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:34,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:34,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:34,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:34,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:34,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:34,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c29c92e[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:34,498 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:34,498 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:34,511 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:34,511 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:34,518 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:34,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:34,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:34,518 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:34,518 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:34,518 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:34,519 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:34,519 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:34,519 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:34,522 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:34,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:34,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:34,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:34,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:34,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:34,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:34,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:34,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:34,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:34,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:34,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:34,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:34,525 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:34,525 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:34,525 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:34,525 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:34,525 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:34,528 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:34,528 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:34,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:34,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:34,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:34,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:34,528 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:34,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:34,529 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:34,530 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:34,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:34,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:34,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:34,530 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:34,530 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:34,530 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:34,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:34,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:34,531 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:34,531 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:34,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:34,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:34,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:34,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:34,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:34,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:34,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:34,532 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:34,534 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:34,534 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:34,534 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:34,534 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:34,534 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:34,534 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:34,534 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:34,631 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:34,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:34,633 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:34,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:34,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:34,650 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:34,650 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:34,650 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:34,650 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:34,652 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:34,653 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:34,654 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:34,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:34,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:34,654 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:34,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:34,655 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:34,655 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:34,655 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:34,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 00ef1cabe628ad083ccaa287ccbbfb01 2023-03-20 10:58:34,655 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 00ef1cabe628ad083ccaa287ccbbfb01 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-20 10:58:34,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:34,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:34,655 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=A 2023-03-20 10:58:34,655 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:34,655 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:34,655 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=B 2023-03-20 10:58:34,655 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:34,656 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=C 2023-03-20 10:58:34,656 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:34,667 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:34,667 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:34,667 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:34,667 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:34,667 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:34,677 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:34,677 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:34,677 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:34,678 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:34,678 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. as already flushing 2023-03-20 10:58:34,678 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:34,678 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:34,678 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:34,678 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:34,678 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:34,678 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:34,681 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:34,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:34,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:34,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:34,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:34,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:34,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:34,682 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:34,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:34,683 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:34,683 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:34,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:34,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:34,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7933968f[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:34,683 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:34,684 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:34,684 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:34,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:34,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:34,684 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:34,684 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:34,684 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:34,685 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:34,685 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:34,685 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:34,685 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:34,686 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:34,686 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:34,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:34,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:34,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:34,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:34,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:34,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:34,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:34,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:34,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:34,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:34,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:34,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:34,688 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:34,688 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:34,688 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:34,688 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:34,688 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:34,692 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:34,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52176 deadline: 1679309974684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:34,693 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:34,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52190 deadline: 1679309974691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:34,695 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:34,695 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:34,695 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:34,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:34,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:34,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:34,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:34,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:34,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:34,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:34,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:34,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:34,697 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:34,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:34,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:34,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:34,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:34,698 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:34,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:34,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:34,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:34,703 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:34,703 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:34,703 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:34,703 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:34,703 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:34,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:34,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:34,703 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:34,703 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:34,703 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:34,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:34,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:34,704 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:34,704 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:34,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:34,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:34,703 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:34,704 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:34,706 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:34,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52164 deadline: 1679309974692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:34,707 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:34,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52180 deadline: 1679309974692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:34,709 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=516 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/A/75c2687799ff4c0282f76e2289308cca 2023-03-20 10:58:34,764 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=516 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/B/523a8a7b45e244d8986be99111857c61 2023-03-20 10:58:34,801 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:34,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52176 deadline: 1679309974794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:34,804 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:34,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:34,805 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:34,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52190 deadline: 1679309974794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:34,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:34,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:34,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:34,818 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:34,818 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:34,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52164 deadline: 1679309974808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:34,818 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:34,818 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:34,818 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:34,819 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:34,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52180 deadline: 1679309974810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:34,822 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:34,822 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:34,823 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:34,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:34,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:34,824 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:34,824 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:34,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:34,824 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:34,825 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:34,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:34,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:34,825 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:34,831 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:34,831 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:34,831 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:34,831 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:34,831 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:34,839 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:34,839 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:34,839 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:34,839 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:34,839 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. as already flushing 2023-03-20 10:58:34,839 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:34,839 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:34,839 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:34,839 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:34,839 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:34,839 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:34,842 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:34,843 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:34,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:34,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:34,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:34,844 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:34,844 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:34,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:34,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:34,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:34,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:34,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:34,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:34,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67df63af[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:34,846 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:34,846 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:34,853 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:34,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:34,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:34,854 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:34,854 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:34,854 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:34,854 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:34,854 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:34,855 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:34,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:34,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:34,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:34,856 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:34,856 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:34,856 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:34,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:34,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:34,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:34,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:34,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:34,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:34,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:34,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:34,860 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:34,860 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:34,860 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:34,861 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:34,861 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:34,861 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:34,863 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:34,863 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:34,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:34,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:34,863 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:34,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:34,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:34,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:34,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:34,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:34,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:34,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:34,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:34,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:34,865 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:34,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:34,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:34,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:34,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:34,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:34,868 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:34,870 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:34,870 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:34,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:34,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:34,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:34,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:34,870 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:34,871 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:34,871 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:34,871 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:34,871 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:34,871 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:34,871 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:34,871 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:34,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:34,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:34,871 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:34,871 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:34,974 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:34,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:34,976 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:34,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:34,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:34,996 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:34,996 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:34,996 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:34,996 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:34,996 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:34,996 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:35,011 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,011 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,011 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:35,011 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:35,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:35,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:35,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:35,013 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:35,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:35,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:35,014 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:35,018 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:35,018 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:35,018 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:35,018 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:35,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52176 deadline: 1679309975003, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,018 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:35,018 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:35,024 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:35,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52190 deadline: 1679309975012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,029 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:35,029 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:35,029 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:35,029 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:35,029 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. as already flushing 2023-03-20 10:58:35,029 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:35,029 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:35,029 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:35,029 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,029 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:35,029 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:35,032 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,032 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:35,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:35,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:35,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:35,033 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:35,033 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:35,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:35,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:35,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:35,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:35,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c5491f8[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:35,034 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:35,034 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:35,035 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:35,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:35,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:35,036 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:35,036 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:35,036 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:35,036 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:35,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52164 deadline: 1679309975021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,036 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:35,037 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:35,037 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:35,037 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,037 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:35,037 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:35,037 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:35,037 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:35,037 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:35,037 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:35,037 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:35,038 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:35,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:35,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:35,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:35,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:35,039 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,039 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:35,039 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:35,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:35,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:35,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:35,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:35,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:35,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:35,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:35,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:35,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,040 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,040 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:35,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:35,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52180 deadline: 1679309975026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:35,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,042 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,042 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:35,042 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:35,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:35,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:35,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:35,042 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:35,042 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:35,042 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:35,043 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:35,043 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:35,042 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:35,043 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:35,043 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:35,043 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:35,043 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:35,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:35,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:35,143 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:35,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:35,148 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:35,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:35,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:35,159 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:35,159 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:35,159 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:35,159 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:35,159 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:35,159 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:35,162 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,162 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,162 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:35,170 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:35,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:35,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:35,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:35,171 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:35,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:35,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:35,171 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:35,172 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:35,172 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:35,172 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:35,172 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:35,172 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:35,172 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:35,172 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:35,172 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:35,172 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:35,172 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. as already flushing 2023-03-20 10:58:35,172 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:35,172 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:35,172 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:35,173 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,173 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:35,173 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:35,173 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:35,173 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:35,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:35,174 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:35,174 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:35,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:35,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:35,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:35,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:35,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:35,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e0659e5[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:35,176 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:35,176 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:35,177 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,177 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:35,178 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:35,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:35,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:35,182 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:35,182 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:35,182 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:35,186 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,186 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:35,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:35,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:35,186 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:35,186 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:35,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:35,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:35,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:35,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:35,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:35,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:35,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:35,187 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:35,187 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:35,187 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:35,187 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:35,188 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:35,188 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:35,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:35,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:35,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:35,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:35,189 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:35,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:35,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:35,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,190 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:35,190 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:35,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:35,190 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:35,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:35,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:35,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:35,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:35,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:35,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,193 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,193 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:35,193 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:35,193 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,193 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:35,193 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:35,193 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:35,193 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:35,193 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:35,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:35,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:35,193 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:35,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:35,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:35,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:35,195 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:35,195 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:35,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:35,203 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=516 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/C/c846580377ac412c8fd5c746716be020 2023-03-20 10:58:35,211 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/A/75c2687799ff4c0282f76e2289308cca as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/75c2687799ff4c0282f76e2289308cca 2023-03-20 10:58:35,232 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/75c2687799ff4c0282f76e2289308cca, entries=200, sequenceid=516, filesize=14.3 K 2023-03-20 10:58:35,233 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/B/523a8a7b45e244d8986be99111857c61 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/523a8a7b45e244d8986be99111857c61 2023-03-20 10:58:35,240 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/523a8a7b45e244d8986be99111857c61, entries=150, sequenceid=516, filesize=11.9 K 2023-03-20 10:58:35,240 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/C/c846580377ac412c8fd5c746716be020 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/c846580377ac412c8fd5c746716be020 2023-03-20 10:58:35,244 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/c846580377ac412c8fd5c746716be020, entries=150, sequenceid=516, filesize=11.9 K 2023-03-20 10:58:35,248 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 00ef1cabe628ad083ccaa287ccbbfb01 in 593ms, sequenceid=516, compaction requested=true 2023-03-20 10:58:35,248 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 00ef1cabe628ad083ccaa287ccbbfb01: 2023-03-20 10:58:35,248 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:58:35,248 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 10:58:35,248 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 10:58:35,248 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:58:35,248 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:58:35,252 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 52138 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 10:58:35,252 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49698 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 10:58:35,252 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1912): 00ef1cabe628ad083ccaa287ccbbfb01/A is initiating minor compaction (all files) 2023-03-20 10:58:35,252 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): 00ef1cabe628ad083ccaa287ccbbfb01/B is initiating minor compaction (all files) 2023-03-20 10:58:35,252 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 00ef1cabe628ad083ccaa287ccbbfb01/A in TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:35,252 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 00ef1cabe628ad083ccaa287ccbbfb01/B in TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:35,252 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/caa40d24be7e40118a1e736f96a63e09, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/3983ea79ff1c4636818fce7330fccbfd, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/18861f4e90f8416a801e645786b8fab0, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/523a8a7b45e244d8986be99111857c61] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp, totalSize=48.5 K 2023-03-20 10:58:35,252 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/09f1d94a719242d6a6ec232a13252f10, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/0a900540558c4a53a06cd7238db3fb57, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/56b61faa6bec4e9c8c49a2794744e0dc, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/75c2687799ff4c0282f76e2289308cca] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp, totalSize=50.9 K 2023-03-20 10:58:35,252 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting caa40d24be7e40118a1e736f96a63e09, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=461, earliestPutTs=1679309912337 2023-03-20 10:58:35,252 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 09f1d94a719242d6a6ec232a13252f10, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=461, earliestPutTs=1679309912337 2023-03-20 10:58:35,252 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 3983ea79ff1c4636818fce7330fccbfd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=481, earliestPutTs=1679309912693 2023-03-20 10:58:35,253 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 0a900540558c4a53a06cd7238db3fb57, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=481, earliestPutTs=1679309912693 2023-03-20 10:58:35,253 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 18861f4e90f8416a801e645786b8fab0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=496, earliestPutTs=1679309913334 2023-03-20 10:58:35,253 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 56b61faa6bec4e9c8c49a2794744e0dc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=496, earliestPutTs=1679309913334 2023-03-20 10:58:35,253 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 523a8a7b45e244d8986be99111857c61, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=516, earliestPutTs=1679309914019 2023-03-20 10:58:35,253 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 75c2687799ff4c0282f76e2289308cca, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=516, earliestPutTs=1679309914013 2023-03-20 10:58:35,290 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): 00ef1cabe628ad083ccaa287ccbbfb01#B#compaction#478 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:58:35,294 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] throttle.PressureAwareThroughputController(145): 00ef1cabe628ad083ccaa287ccbbfb01#A#compaction#479 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:58:35,295 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:35,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:35,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:35,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:35,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:35,307 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:35,307 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:35,307 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:35,308 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:35,308 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:35,308 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:35,309 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:35,309 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:35,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:35,309 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,309 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:35,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:35,309 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:35,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:35,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:35,310 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:35,328 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:35,328 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:35,328 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:35,328 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:35,328 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:35,329 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:35,329 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:35,329 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:35,329 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:35,329 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 00ef1cabe628ad083ccaa287ccbbfb01 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-20 10:58:35,329 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. as already flushing 2023-03-20 10:58:35,329 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=A 2023-03-20 10:58:35,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 00ef1cabe628ad083ccaa287ccbbfb01 2023-03-20 10:58:35,329 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:35,329 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=B 2023-03-20 10:58:35,330 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:35,330 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=C 2023-03-20 10:58:35,330 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:35,366 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/B/c3ac0df799f14e8197beaf4a00bceb2a as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/c3ac0df799f14e8197beaf4a00bceb2a 2023-03-20 10:58:35,373 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 00ef1cabe628ad083ccaa287ccbbfb01/B of 00ef1cabe628ad083ccaa287ccbbfb01 into c3ac0df799f14e8197beaf4a00bceb2a(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:58:35,373 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 00ef1cabe628ad083ccaa287ccbbfb01: 2023-03-20 10:58:35,373 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01., storeName=00ef1cabe628ad083ccaa287ccbbfb01/B, priority=12, startTime=1679309915248; duration=0sec 2023-03-20 10:58:35,374 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:58:35,374 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 10:58:35,376 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49664 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 10:58:35,377 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): 00ef1cabe628ad083ccaa287ccbbfb01/C is initiating minor compaction (all files) 2023-03-20 10:58:35,377 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 00ef1cabe628ad083ccaa287ccbbfb01/C in TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:35,377 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/cc2521458e4c45db817d9e61eabd2d0e, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/665612b9488a4c8f901ec084028c3314, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/b292772694454a79a0d284bec3be81d4, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/c846580377ac412c8fd5c746716be020] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp, totalSize=48.5 K 2023-03-20 10:58:35,377 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting cc2521458e4c45db817d9e61eabd2d0e, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=461, earliestPutTs=1679309912337 2023-03-20 10:58:35,377 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 665612b9488a4c8f901ec084028c3314, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=481, earliestPutTs=1679309912693 2023-03-20 10:58:35,378 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting b292772694454a79a0d284bec3be81d4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=496, earliestPutTs=1679309913334 2023-03-20 10:58:35,378 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting c846580377ac412c8fd5c746716be020, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=516, earliestPutTs=1679309914019 2023-03-20 10:58:35,381 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/A/c293b7859f4249859903025e24196204 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/c293b7859f4249859903025e24196204 2023-03-20 10:58:35,382 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=529 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/A/79cc21c1fa634956ad9dc8e87b27e480 2023-03-20 10:58:35,398 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 00ef1cabe628ad083ccaa287ccbbfb01/A of 00ef1cabe628ad083ccaa287ccbbfb01 into c293b7859f4249859903025e24196204(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:58:35,398 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 00ef1cabe628ad083ccaa287ccbbfb01: 2023-03-20 10:58:35,398 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01., storeName=00ef1cabe628ad083ccaa287ccbbfb01/A, priority=12, startTime=1679309915248; duration=0sec 2023-03-20 10:58:35,398 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:58:35,400 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:35,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52180 deadline: 1679309975387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,406 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:35,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52190 deadline: 1679309975396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,411 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): 00ef1cabe628ad083ccaa287ccbbfb01#C#compaction#481 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:58:35,413 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:35,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52164 deadline: 1679309975400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,415 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:35,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52176 deadline: 1679309975402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,431 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=529 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/B/68adbb5b4f9f4db0b19bcbf2155995bf 2023-03-20 10:58:35,448 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/C/e3ae9764d49f413391052fe25f16823b as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/e3ae9764d49f413391052fe25f16823b 2023-03-20 10:58:35,462 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 00ef1cabe628ad083ccaa287ccbbfb01/C of 00ef1cabe628ad083ccaa287ccbbfb01 into e3ae9764d49f413391052fe25f16823b(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:58:35,462 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 00ef1cabe628ad083ccaa287ccbbfb01: 2023-03-20 10:58:35,462 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01., storeName=00ef1cabe628ad083ccaa287ccbbfb01/C, priority=12, startTime=1679309915248; duration=0sec 2023-03-20 10:58:35,462 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:58:35,477 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=529 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/C/0426c1c54e414f00bdc8d8db5770348e 2023-03-20 10:58:35,495 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/A/79cc21c1fa634956ad9dc8e87b27e480 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/79cc21c1fa634956ad9dc8e87b27e480 2023-03-20 10:58:35,501 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/79cc21c1fa634956ad9dc8e87b27e480, entries=150, sequenceid=529, filesize=11.9 K 2023-03-20 10:58:35,502 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/B/68adbb5b4f9f4db0b19bcbf2155995bf as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/68adbb5b4f9f4db0b19bcbf2155995bf 2023-03-20 10:58:35,507 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/68adbb5b4f9f4db0b19bcbf2155995bf, entries=150, sequenceid=529, filesize=11.9 K 2023-03-20 10:58:35,508 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/C/0426c1c54e414f00bdc8d8db5770348e as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/0426c1c54e414f00bdc8d8db5770348e 2023-03-20 10:58:35,510 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:35,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52180 deadline: 1679309975502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,515 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:35,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52190 deadline: 1679309975507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,519 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/0426c1c54e414f00bdc8d8db5770348e, entries=150, sequenceid=529, filesize=11.9 K 2023-03-20 10:58:35,519 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for 00ef1cabe628ad083ccaa287ccbbfb01 in 190ms, sequenceid=529, compaction requested=false 2023-03-20 10:58:35,519 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 00ef1cabe628ad083ccaa287ccbbfb01: 2023-03-20 10:58:35,520 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:35,520 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:35,520 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:35,520 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,520 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:35,520 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:35,522 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:35,523 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:35,523 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:35,526 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:35,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:35,528 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679309915365, max timestamp=1679309915525Num uniques 859; 2023-03-20 10:58:35,528 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-20 10:58:35,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 00ef1cabe628ad083ccaa287ccbbfb01 2023-03-20 10:58:35,529 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 00ef1cabe628ad083ccaa287ccbbfb01 3/3 column families, dataSize=89.45 KB heapSize=262.11 KB 2023-03-20 10:58:35,531 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679309915365, max timestamp=1679309915525Num uniques 859; 2023-03-20 10:58:35,531 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-20 10:58:35,533 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679309915365, max timestamp=1679309915525Num uniques 859; 2023-03-20 10:58:35,533 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-20 10:58:35,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:35,538 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=A 2023-03-20 10:58:35,538 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:58:35,538 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=B 2023-03-20 10:58:35,538 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:58:35,538 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=C 2023-03-20 10:58:35,538 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:58:35,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:35,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:35,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:35,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:35,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b642744[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:35,557 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:35,557 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:35,558 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:35,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:35,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:35,558 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:35,558 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:35,558 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:35,558 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,558 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:35,559 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:35,560 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:35,559 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=554 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/A/a2f1cb0f51264a0696fa9d2be1a93725 2023-03-20 10:58:35,560 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:35,566 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:35,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:35,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:35,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:35,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:35,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:35,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:35,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:35,570 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:35,570 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:35,570 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:35,570 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:35,570 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:35,570 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:35,572 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:35,572 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:35,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:35,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:35,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:35,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:35,572 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:35,573 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,574 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:35,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:35,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:35,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:35,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:35,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:35,576 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:35,576 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:35,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:35,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:35,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:35,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:35,577 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:35,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:35,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:35,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:35,577 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:35,577 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:35,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:35,578 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:35,582 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,582 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:35,582 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:35,582 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,582 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:35,582 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:35,582 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:35,629 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679309915591, max timestamp=1679309915627Num uniques 859; 2023-03-20 10:58:35,629 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679309915591, max timestamp=1679309915627Num uniques 859; 2023-03-20 10:58:35,629 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-20 10:58:35,629 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679309915591, max timestamp=1679309915627Num uniques 859; 2023-03-20 10:58:35,629 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-20 10:58:35,629 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-20 10:58:35,677 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:35,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:35,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:35,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:35,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:35,681 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:35,681 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:35,681 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:35,681 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:35,681 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:35,681 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:35,682 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:35,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:35,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:35,682 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:35,682 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:35,683 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,683 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:35,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:35,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:35,683 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:35,683 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:35,683 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:35,683 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:35,683 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:35,683 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:35,683 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:35,684 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:35,684 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:35,684 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:35,684 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. as already flushing 2023-03-20 10:58:35,684 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:35,684 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:35,684 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:35,684 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,684 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:35,684 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:35,686 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:35,687 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:35,687 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:35,687 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:35,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:35,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:35,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:35,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:35,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:35,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:35,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47baf59c[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:35,689 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:35,689 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:35,689 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:35,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:35,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:35,689 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:35,689 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:35,690 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:35,690 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,690 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:35,690 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:35,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:35,690 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:35,691 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:35,691 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:35,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:35,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:35,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:35,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:35,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:35,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:35,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:35,693 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:35,693 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:35,693 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:35,693 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:35,693 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:35,694 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:35,694 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:35,694 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:35,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:35,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:35,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:35,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:35,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:35,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:35,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:35,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:35,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:35,696 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:35,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:35,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:35,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,697 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:35,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:35,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,699 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,699 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:35,699 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:35,699 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,699 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:35,699 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:35,699 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:35,699 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:35,699 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:35,699 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:35,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:35,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:35,701 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:35,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:35,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:35,701 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:35,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:35,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:35,714 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:35,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52164 deadline: 1679309975704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,716 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679309915671, max timestamp=1679309915713Num uniques 1118; 2023-03-20 10:58:35,716 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=7688, suffix heap size=144240,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=168 2023-03-20 10:58:35,720 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679309915671, max timestamp=1679309915713Num uniques 1118; 2023-03-20 10:58:35,720 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=7688, suffix heap size=144240,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=168 2023-03-20 10:58:35,721 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679309915671, max timestamp=1679309915713Num uniques 1118; 2023-03-20 10:58:35,721 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=7688, suffix heap size=144240,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=168 2023-03-20 10:58:35,800 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1679309915736, max timestamp=1679309915797Num uniques 1027; 2023-03-20 10:58:35,800 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-03-20 10:58:35,801 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:35,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:35,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:35,804 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8097, totalHeapSize=18648, min timestamp=1679309915773, max timestamp=1679309915800Num uniques 1027; 2023-03-20 10:58:35,804 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=47027, new segment data size=8097, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-03-20 10:58:35,801 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1679309915736, max timestamp=1679309915797Num uniques 1027; 2023-03-20 10:58:35,806 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-03-20 10:58:35,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:35,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:35,807 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:35,807 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:35,807 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:35,807 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:35,807 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:35,807 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:35,808 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,808 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:35,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:35,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:35,808 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,808 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:35,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:35,809 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:35,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:35,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:35,809 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:35,825 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:35,825 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:35,825 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:35,825 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:35,825 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:35,832 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:35,832 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:35,832 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:35,832 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:35,832 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. as already flushing 2023-03-20 10:58:35,832 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:35,832 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:35,832 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:35,832 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,832 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:35,832 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:35,834 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,834 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:35,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:35,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:35,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:35,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:35,835 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:35,835 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:35,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:35,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:35,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:35,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f4977c2[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:35,836 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:35,836 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:35,837 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:35,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:35,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:35,838 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:35,838 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:35,838 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:35,838 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,838 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:35,840 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:35,840 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:35,840 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:35,840 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:35,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:35,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:35,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:35,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:35,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:35,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:35,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:35,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:35,847 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:35,847 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:35,847 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:35,847 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:35,847 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:35,877 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:35,878 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:35,878 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:35,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:35,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:35,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:35,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:35,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:35,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:35,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:35,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:35,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:35,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:35,884 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1679309915840, max timestamp=1679309915876Num uniques 1086; 2023-03-20 10:58:35,884 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-03-20 10:58:35,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:35,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:35,887 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,887 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1679309915840, max timestamp=1679309915876Num uniques 1086; 2023-03-20 10:58:35,887 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-03-20 10:58:35,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:35,888 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1679309915840, max timestamp=1679309915882Num uniques 1036; 2023-03-20 10:58:35,889 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-03-20 10:58:35,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:35,892 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,894 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:35,894 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:35,894 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:35,894 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:35,894 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:35,895 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:35,894 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,894 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:35,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:35,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:35,895 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:35,895 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:35,895 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:35,895 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:35,895 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:35,895 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:35,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:35,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:35,942 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1679309915912, max timestamp=1679309915939Num uniques 1095; 2023-03-20 10:58:35,942 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=13510, suffix heap size=141848,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=295 2023-03-20 10:58:35,942 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1679309915912, max timestamp=1679309915939Num uniques 1095; 2023-03-20 10:58:35,942 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=13510, suffix heap size=141848,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=295 2023-03-20 10:58:35,944 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=295, cellSize=13510, totalHeapSize=30920, min timestamp=1679309915912, max timestamp=1679309915939Num uniques 1095; 2023-03-20 10:58:35,944 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=13510, suffix heap size=141848,new segment heap size=30920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=295 2023-03-20 10:58:35,994 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=554 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/B/c376fa9bc3c94c088cd5b6d14cdfa700 2023-03-20 10:58:35,996 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:35,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:36,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:36,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:36,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:36,007 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:36,008 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:36,008 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:36,008 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:36,014 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:36,014 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:36,015 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,015 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:36,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:36,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:36,015 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,015 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:36,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:36,016 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:36,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:36,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:36,016 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:36,027 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9334, totalHeapSize=21456, min timestamp=1679309915981, max timestamp=1679309916004Num uniques 1154; 2023-03-20 10:58:36,027 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=52844, new segment data size=9334, suffix heap size=147984,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=204 2023-03-20 10:58:36,028 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9334, totalHeapSize=21456, min timestamp=1679309915981, max timestamp=1679309916004Num uniques 1154; 2023-03-20 10:58:36,028 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=52844, new segment data size=9334, suffix heap size=147984,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=204 2023-03-20 10:58:36,029 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9334, totalHeapSize=21456, min timestamp=1679309915981, max timestamp=1679309916004Num uniques 1154; 2023-03-20 10:58:36,029 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=52844, new segment data size=9334, suffix heap size=147984,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1154, new segment cells count=204 2023-03-20 10:58:36,029 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:36,029 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:36,029 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:36,029 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:36,029 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:36,035 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:36,035 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:36,035 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:36,035 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:36,036 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. as already flushing 2023-03-20 10:58:36,036 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:36,036 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:36,036 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:36,036 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,036 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:36,036 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:36,037 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=554 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/C/979d99d0f85b4090bb3c064e5eca9c0f 2023-03-20 10:58:36,037 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:36,037 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:36,038 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:36,042 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:36,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:36,044 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/A/a2f1cb0f51264a0696fa9d2be1a93725 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/a2f1cb0f51264a0696fa9d2be1a93725 2023-03-20 10:58:36,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:36,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:36,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:36,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:36,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:36,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@630cc2bf[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:36,049 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:36,049 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:36,050 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:36,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:36,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:36,050 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:36,050 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:36,050 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,050 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:36,050 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:36,051 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/a2f1cb0f51264a0696fa9d2be1a93725, entries=200, sequenceid=554, filesize=14.3 K 2023-03-20 10:58:36,051 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,051 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:36,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:36,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:36,051 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:36,051 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:36,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:36,052 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/B/c376fa9bc3c94c088cd5b6d14cdfa700 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/c376fa9bc3c94c088cd5b6d14cdfa700 2023-03-20 10:58:36,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:36,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:36,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:36,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:36,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:36,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:36,054 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:36,054 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:36,054 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:36,054 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:36,054 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:36,056 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/c376fa9bc3c94c088cd5b6d14cdfa700, entries=150, sequenceid=554, filesize=11.9 K 2023-03-20 10:58:36,057 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/C/979d99d0f85b4090bb3c064e5eca9c0f as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/979d99d0f85b4090bb3c064e5eca9c0f 2023-03-20 10:58:36,057 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:36,057 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:36,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:36,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:36,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:36,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:36,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:36,057 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:36,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:36,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:36,059 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:36,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:36,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:36,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:36,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:36,061 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:36,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:36,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,065 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/979d99d0f85b4090bb3c064e5eca9c0f, entries=150, sequenceid=554, filesize=11.9 K 2023-03-20 10:58:36,067 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~86.11 KB/88176, currentSize=87.22 KB/89310 for 00ef1cabe628ad083ccaa287ccbbfb01 in 538ms, sequenceid=554, compaction requested=true 2023-03-20 10:58:36,067 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 00ef1cabe628ad083ccaa287ccbbfb01: 2023-03-20 10:58:36,067 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:58:36,067 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-20 10:58:36,067 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-20 10:58:36,067 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 10:58:36,067 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 10:58:36,068 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40088 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 10:58:36,068 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): 00ef1cabe628ad083ccaa287ccbbfb01/A is initiating minor compaction (all files) 2023-03-20 10:58:36,069 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 00ef1cabe628ad083ccaa287ccbbfb01/A in TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:36,069 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/c293b7859f4249859903025e24196204, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/79cc21c1fa634956ad9dc8e87b27e480, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/a2f1cb0f51264a0696fa9d2be1a93725] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp, totalSize=39.1 K 2023-03-20 10:58:36,069 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37648 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 10:58:36,069 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1912): 00ef1cabe628ad083ccaa287ccbbfb01/B is initiating minor compaction (all files) 2023-03-20 10:58:36,069 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 00ef1cabe628ad083ccaa287ccbbfb01/B in TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:36,069 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/c3ac0df799f14e8197beaf4a00bceb2a, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/68adbb5b4f9f4db0b19bcbf2155995bf, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/c376fa9bc3c94c088cd5b6d14cdfa700] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp, totalSize=36.8 K 2023-03-20 10:58:36,070 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting c293b7859f4249859903025e24196204, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=516, earliestPutTs=1679309914019 2023-03-20 10:58:36,070 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:36,070 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:36,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:36,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:36,070 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,071 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:36,071 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:36,071 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:36,071 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:36,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:36,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:36,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:36,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:36,071 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,071 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 79cc21c1fa634956ad9dc8e87b27e480, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=529, earliestPutTs=1679309914677 2023-03-20 10:58:36,071 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:36,071 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:36,071 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:36,071 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:36,071 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:36,071 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting c3ac0df799f14e8197beaf4a00bceb2a, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=516, earliestPutTs=1679309914019 2023-03-20 10:58:36,071 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting a2f1cb0f51264a0696fa9d2be1a93725, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=554, earliestPutTs=1679309915365 2023-03-20 10:58:36,072 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 68adbb5b4f9f4db0b19bcbf2155995bf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=529, earliestPutTs=1679309914677 2023-03-20 10:58:36,072 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting c376fa9bc3c94c088cd5b6d14cdfa700, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=554, earliestPutTs=1679309915399 2023-03-20 10:58:36,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 00ef1cabe628ad083ccaa287ccbbfb01 2023-03-20 10:58:36,074 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 00ef1cabe628ad083ccaa287ccbbfb01 3/3 column families, dataSize=93.93 KB heapSize=261.23 KB 2023-03-20 10:58:36,074 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=A 2023-03-20 10:58:36,074 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:58:36,074 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=B 2023-03-20 10:58:36,074 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:58:36,075 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=C 2023-03-20 10:58:36,075 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:58:36,102 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): 00ef1cabe628ad083ccaa287ccbbfb01#A#compaction#488 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:58:36,106 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] throttle.PressureAwareThroughputController(145): 00ef1cabe628ad083ccaa287ccbbfb01#B#compaction#489 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:58:36,130 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/B/fb375bafdae141db9c515ba5e7013234 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/fb375bafdae141db9c515ba5e7013234 2023-03-20 10:58:36,150 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 00ef1cabe628ad083ccaa287ccbbfb01/B of 00ef1cabe628ad083ccaa287ccbbfb01 into fb375bafdae141db9c515ba5e7013234(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:58:36,150 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 00ef1cabe628ad083ccaa287ccbbfb01: 2023-03-20 10:58:36,150 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01., storeName=00ef1cabe628ad083ccaa287ccbbfb01/B, priority=13, startTime=1679309916067; duration=0sec 2023-03-20 10:58:36,150 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:58:36,150 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:36,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52176 deadline: 1679309976109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,150 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 10:58:36,159 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37614 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 10:58:36,159 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1912): 00ef1cabe628ad083ccaa287ccbbfb01/C is initiating minor compaction (all files) 2023-03-20 10:58:36,159 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 00ef1cabe628ad083ccaa287ccbbfb01/C in TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:36,159 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/e3ae9764d49f413391052fe25f16823b, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/0426c1c54e414f00bdc8d8db5770348e, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/979d99d0f85b4090bb3c064e5eca9c0f] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp, totalSize=36.7 K 2023-03-20 10:58:36,160 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting e3ae9764d49f413391052fe25f16823b, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=516, earliestPutTs=1679309914019 2023-03-20 10:58:36,160 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 0426c1c54e414f00bdc8d8db5770348e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=529, earliestPutTs=1679309914677 2023-03-20 10:58:36,160 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 979d99d0f85b4090bb3c064e5eca9c0f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=554, earliestPutTs=1679309915399 2023-03-20 10:58:36,167 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] throttle.PressureAwareThroughputController(145): 00ef1cabe628ad083ccaa287ccbbfb01#C#compaction#490 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:58:36,171 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:36,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52164 deadline: 1679309976151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,171 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:36,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52190 deadline: 1679309976151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,171 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:36,171 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:36,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 307 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52180 deadline: 1679309976151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:36,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:36,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:36,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:36,176 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:36,176 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:36,177 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:36,177 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:36,177 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:36,177 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:36,178 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,178 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:36,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:36,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:36,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:36,178 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:36,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:36,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:36,179 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:36,179 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:36,179 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:36,179 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,179 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:36,179 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:36,179 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:36,179 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:36,182 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:36,182 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:36,182 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:36,182 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:36,182 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. as already flushing 2023-03-20 10:58:36,182 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:36,182 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:36,182 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:36,182 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,182 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:36,182 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:36,185 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:36,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:36,185 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:36,186 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:36,186 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:36,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:36,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:36,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:36,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:36,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:36,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1928879d[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:36,188 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:36,189 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:36,189 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,189 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:36,192 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:36,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:36,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:36,192 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:36,192 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:36,192 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:36,193 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:36,193 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:36,193 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:36,193 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:36,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:36,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:36,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:36,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:36,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:36,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:36,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:36,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:36,196 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:36,196 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:36,196 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:36,196 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:36,196 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:36,197 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:36,197 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:36,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:36,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:36,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:36,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:36,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:36,197 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:36,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:36,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:36,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:36,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:36,198 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:36,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:36,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:36,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:36,199 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:36,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,211 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:36,211 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:36,211 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:36,211 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:36,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:36,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:36,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:36,211 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:36,211 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:36,211 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,211 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:36,211 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:36,211 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:36,212 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:36,212 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:36,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:36,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:36,259 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:36,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52176 deadline: 1679309976252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,286 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:36,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52190 deadline: 1679309976273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,287 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:36,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52164 deadline: 1679309976274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,287 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:36,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 309 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52180 deadline: 1679309976274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,296 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:36,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:52136 deadline: 1679309976296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,299 DEBUG [Thread-2963] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18270 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01., hostname=jenkins-hbase20.apache.org,36041,1679309777062, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-20 10:58:36,312 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:36,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:36,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:36,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:36,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:36,314 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:36,314 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:36,314 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:36,314 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:36,315 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:36,315 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:36,315 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:36,315 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:36,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:36,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:36,316 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,316 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:36,316 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:36,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:36,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:36,318 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:36,318 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:36,318 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:36,318 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:36,318 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:36,318 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:36,318 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:36,318 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:36,318 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:36,318 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:36,319 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. as already flushing 2023-03-20 10:58:36,319 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:36,319 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:36,319 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:36,319 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,319 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:36,319 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:36,320 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,320 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:36,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:36,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:36,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:36,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:36,321 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:36,321 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:36,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:36,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:36,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:36,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9ae9731[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:36,321 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:36,321 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:36,322 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:36,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:36,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:36,322 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:36,322 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:36,322 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,322 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:36,322 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:36,323 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:36,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:36,323 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:36,323 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:36,323 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:36,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:36,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:36,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:36,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:36,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:36,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:36,325 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:36,325 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:36,325 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:36,325 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:36,325 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:36,325 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:36,336 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:36,336 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:36,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:36,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:36,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:36,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:36,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:36,336 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:36,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:36,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:36,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:36,336 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:36,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:36,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:36,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:36,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,338 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:36,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:36,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,344 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:36,345 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:36,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:36,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:36,345 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:36,345 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:36,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:36,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:36,345 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,345 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:36,345 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:36,345 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,345 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:36,345 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:36,345 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:36,345 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:36,345 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:36,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:36,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:36,445 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:36,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:36,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:36,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:36,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:36,448 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:36,448 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:36,448 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:36,448 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:36,448 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:36,448 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:36,449 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:36,449 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:36,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:36,449 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,449 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:36,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:36,450 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:36,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:36,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:36,450 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:36,451 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:36,451 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:36,451 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:36,451 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:36,451 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:36,451 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:36,451 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:36,451 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:36,451 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:36,452 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. as already flushing 2023-03-20 10:58:36,452 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:36,452 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:36,452 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:36,452 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,452 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:36,452 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:36,452 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:36,452 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:36,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:36,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:36,453 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:36,453 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:36,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:36,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:36,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:36,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:36,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d7d06a1[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:36,454 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:36,454 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:36,454 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:36,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:36,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:36,454 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:36,455 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:36,455 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:36,455 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,455 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:36,455 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,455 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:36,455 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:36,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:36,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:36,455 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:36,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:36,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:36,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:36,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:36,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:36,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:36,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:36,457 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:36,457 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:36,457 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:36,457 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:36,457 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:36,458 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:36,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:36,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:36,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:36,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:36,458 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,458 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:36,458 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:36,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:36,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:36,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:36,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:36,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:36,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:36,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:36,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:36,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,460 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:36,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:36,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,462 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,462 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:36,463 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:36,462 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:36,463 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,463 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:36,463 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:36,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:36,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:36,463 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:36,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:36,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:36,463 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:36,463 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:36,463 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:36,463 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:36,463 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:36,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:36,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:36,468 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:36,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52176 deadline: 1679309976461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,501 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:36,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52164 deadline: 1679309976488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,501 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:36,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 311 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52180 deadline: 1679309976488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,502 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:36,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52190 deadline: 1679309976489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,516 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=670 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/A/72ac9f7690124fd6a817a0becdb222a1 2023-03-20 10:58:36,537 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/A/536cf75feb5243998bb57de793a3eb72 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/536cf75feb5243998bb57de793a3eb72 2023-03-20 10:58:36,551 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 00ef1cabe628ad083ccaa287ccbbfb01/A of 00ef1cabe628ad083ccaa287ccbbfb01 into 536cf75feb5243998bb57de793a3eb72(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:58:36,551 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 00ef1cabe628ad083ccaa287ccbbfb01: 2023-03-20 10:58:36,551 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01., storeName=00ef1cabe628ad083ccaa287ccbbfb01/A, priority=13, startTime=1679309916067; duration=0sec 2023-03-20 10:58:36,551 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:58:36,563 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:36,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:36,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:36,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:36,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:36,576 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:36,576 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:36,576 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:36,576 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:36,576 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:36,576 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:36,577 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:36,577 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:36,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:36,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:36,577 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,577 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:36,578 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:36,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:36,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:36,578 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:36,588 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/C/326b51c674924e818dcc27ae1ccde1a9 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/326b51c674924e818dcc27ae1ccde1a9 2023-03-20 10:58:36,597 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:36,597 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 00ef1cabe628ad083ccaa287ccbbfb01/C of 00ef1cabe628ad083ccaa287ccbbfb01 into 326b51c674924e818dcc27ae1ccde1a9(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:58:36,597 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:36,598 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:36,599 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:36,599 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:36,602 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:36,602 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:36,602 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:36,602 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. started... 2023-03-20 10:58:36,602 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. as already flushing 2023-03-20 10:58:36,602 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:36,602 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:36,602 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:36,602 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,602 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:36,602 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:36,604 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:36,604 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:36,604 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:36,597 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 00ef1cabe628ad083ccaa287ccbbfb01: 2023-03-20 10:58:36,604 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,604 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01., storeName=00ef1cabe628ad083ccaa287ccbbfb01/C, priority=13, startTime=1679309916067; duration=0sec 2023-03-20 10:58:36,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:36,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:36,604 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:58:36,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:36,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:36,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:36,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:36,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:36,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28e1208b[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:36,606 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:36,606 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:36,606 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:36,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:36,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:36,606 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:36,607 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:36,607 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,607 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:36,607 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:36,607 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:36,607 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:36,607 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:36,608 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:36,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:36,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:36,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:36,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:36,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:36,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:36,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:36,610 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:36,610 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:36,610 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:36,610 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:36,610 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:36,610 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:36,611 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:36,611 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:36,611 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:36,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:36,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:36,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:36,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:36,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:36,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:36,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:36,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:36,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:36,612 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:36,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:36,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:36,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,614 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:36,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:36,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,615 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:36,615 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:36,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:36,615 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:36,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:36,615 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,616 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:36,616 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:36,616 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,616 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:36,616 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:36,616 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:36,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:36,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:36,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:36,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:36,616 DEBUG [Thread-2965] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:36,616 DEBUG [Thread-2965] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:36,714 DEBUG [Thread-2966] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0f4b7807 to 127.0.0.1:60226 2023-03-20 10:58:36,714 DEBUG [Thread-2966] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 10:58:36,717 DEBUG [Thread-2968] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x31fc0e3f to 127.0.0.1:60226 2023-03-20 10:58:36,717 DEBUG [Thread-2968] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 10:58:36,717 DEBUG [Thread-2965] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:36,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:36,723 DEBUG [Thread-2972] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0e56ea94 to 127.0.0.1:60226 2023-03-20 10:58:36,723 DEBUG [Thread-2972] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 10:58:36,725 DEBUG [Thread-2970] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x79db57d4 to 127.0.0.1:60226 2023-03-20 10:58:36,725 DEBUG [Thread-2970] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 10:58:36,725 DEBUG [Thread-2974] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4486b3c7 to 127.0.0.1:60226 2023-03-20 10:58:36,725 DEBUG [Thread-2974] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 10:58:36,770 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:36,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52176 deadline: 1679309976770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,802 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:36,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52164 deadline: 1679309976802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,804 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:36,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 313 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52180 deadline: 1679309976803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:36,804 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:36,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:52190 deadline: 1679309976804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:37,006 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=670 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/B/a9b1f1a1ba664ae885192b94e2b71e49 2023-03-20 10:58:37,049 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=670 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/C/b0a463f587544d0c8bbc11c51bb06f29 2023-03-20 10:58:37,065 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/A/72ac9f7690124fd6a817a0becdb222a1 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/72ac9f7690124fd6a817a0becdb222a1 2023-03-20 10:58:37,072 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/72ac9f7690124fd6a817a0becdb222a1, entries=250, sequenceid=670, filesize=16.7 K 2023-03-20 10:58:37,073 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/B/a9b1f1a1ba664ae885192b94e2b71e49 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/a9b1f1a1ba664ae885192b94e2b71e49 2023-03-20 10:58:37,094 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/a9b1f1a1ba664ae885192b94e2b71e49, entries=150, sequenceid=670, filesize=11.9 K 2023-03-20 10:58:37,094 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/C/b0a463f587544d0c8bbc11c51bb06f29 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/b0a463f587544d0c8bbc11c51bb06f29 2023-03-20 10:58:37,098 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/b0a463f587544d0c8bbc11c51bb06f29, entries=150, sequenceid=670, filesize=11.9 K 2023-03-20 10:58:37,099 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~261.19 KB/267456, currentSize=87.22 KB/89310 for 00ef1cabe628ad083ccaa287ccbbfb01 in 1025ms, sequenceid=670, compaction requested=false 2023-03-20 10:58:37,099 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 00ef1cabe628ad083ccaa287ccbbfb01: 2023-03-20 10:58:37,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 00ef1cabe628ad083ccaa287ccbbfb01 2023-03-20 10:58:37,301 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 00ef1cabe628ad083ccaa287ccbbfb01 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-20 10:58:37,302 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=A 2023-03-20 10:58:37,302 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:37,302 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=B 2023-03-20 10:58:37,302 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:37,302 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=C 2023-03-20 10:58:37,302 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:37,302 DEBUG [Thread-2959] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x41ccedcb to 127.0.0.1:60226 2023-03-20 10:58:37,303 DEBUG [Thread-2959] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 10:58:37,319 DEBUG [Thread-2955] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x02e8c6f6 to 127.0.0.1:60226 2023-03-20 10:58:37,319 DEBUG [Thread-2955] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 10:58:37,321 DEBUG [Thread-2961] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x40e6f624 to 127.0.0.1:60226 2023-03-20 10:58:37,321 DEBUG [Thread-2961] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 10:58:37,321 DEBUG [Thread-2957] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2cb7a788 to 127.0.0.1:60226 2023-03-20 10:58:37,321 DEBUG [Thread-2957] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 10:58:37,367 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=690 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/A/f580d93e4e36496fac178d72fa0fdb38 2023-03-20 10:58:37,810 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=690 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/B/1af1575a18554efcbdb6a1828ffe8f79 2023-03-20 10:58:38,248 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=690 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/C/364f078934d743daba9057b04ef90d70 2023-03-20 10:58:38,252 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/A/f580d93e4e36496fac178d72fa0fdb38 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/f580d93e4e36496fac178d72fa0fdb38 2023-03-20 10:58:38,255 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/f580d93e4e36496fac178d72fa0fdb38, entries=150, sequenceid=690, filesize=11.9 K 2023-03-20 10:58:38,255 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/B/1af1575a18554efcbdb6a1828ffe8f79 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/1af1575a18554efcbdb6a1828ffe8f79 2023-03-20 10:58:38,259 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/1af1575a18554efcbdb6a1828ffe8f79, entries=150, sequenceid=690, filesize=11.9 K 2023-03-20 10:58:38,259 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/C/364f078934d743daba9057b04ef90d70 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/364f078934d743daba9057b04ef90d70 2023-03-20 10:58:38,262 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/364f078934d743daba9057b04ef90d70, entries=150, sequenceid=690, filesize=11.9 K 2023-03-20 10:58:38,263 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=20.13 KB/20610 for 00ef1cabe628ad083ccaa287ccbbfb01 in 962ms, sequenceid=690, compaction requested=true 2023-03-20 10:58:38,263 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 00ef1cabe628ad083ccaa287ccbbfb01: 2023-03-20 10:58:38,263 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:58:38,263 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:58:38,263 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 10:58:38,263 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 10:58:38,263 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:58:38,264 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42630 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 10:58:38,264 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37750 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 10:58:38,264 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1912): 00ef1cabe628ad083ccaa287ccbbfb01/A is initiating minor compaction (all files) 2023-03-20 10:58:38,264 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): 00ef1cabe628ad083ccaa287ccbbfb01/B is initiating minor compaction (all files) 2023-03-20 10:58:38,264 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 00ef1cabe628ad083ccaa287ccbbfb01/A in TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:38,264 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 00ef1cabe628ad083ccaa287ccbbfb01/B in TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:38,264 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/536cf75feb5243998bb57de793a3eb72, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/72ac9f7690124fd6a817a0becdb222a1, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/f580d93e4e36496fac178d72fa0fdb38] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp, totalSize=41.6 K 2023-03-20 10:58:38,264 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/fb375bafdae141db9c515ba5e7013234, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/a9b1f1a1ba664ae885192b94e2b71e49, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/1af1575a18554efcbdb6a1828ffe8f79] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp, totalSize=36.9 K 2023-03-20 10:58:38,265 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 536cf75feb5243998bb57de793a3eb72, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=554, earliestPutTs=1679309915399 2023-03-20 10:58:38,265 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting fb375bafdae141db9c515ba5e7013234, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=554, earliestPutTs=1679309915399 2023-03-20 10:58:38,265 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 72ac9f7690124fd6a817a0becdb222a1, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=670, earliestPutTs=1679309916004 2023-03-20 10:58:38,265 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting a9b1f1a1ba664ae885192b94e2b71e49, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=670, earliestPutTs=1679309916057 2023-03-20 10:58:38,265 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting f580d93e4e36496fac178d72fa0fdb38, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=690, earliestPutTs=1679309916106 2023-03-20 10:58:38,265 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 1af1575a18554efcbdb6a1828ffe8f79, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=690, earliestPutTs=1679309916106 2023-03-20 10:58:38,270 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): 00ef1cabe628ad083ccaa287ccbbfb01#B#compaction#496 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:58:38,272 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] throttle.PressureAwareThroughputController(145): 00ef1cabe628ad083ccaa287ccbbfb01#A#compaction#497 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:58:38,677 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/B/bb17460e81db421881c7e9c97b0c919f as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/bb17460e81db421881c7e9c97b0c919f 2023-03-20 10:58:38,679 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/A/bb99d7201d054b5b971434e1afc853f2 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/bb99d7201d054b5b971434e1afc853f2 2023-03-20 10:58:38,681 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 00ef1cabe628ad083ccaa287ccbbfb01/B of 00ef1cabe628ad083ccaa287ccbbfb01 into bb17460e81db421881c7e9c97b0c919f(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:58:38,681 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 00ef1cabe628ad083ccaa287ccbbfb01: 2023-03-20 10:58:38,681 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01., storeName=00ef1cabe628ad083ccaa287ccbbfb01/B, priority=13, startTime=1679309918263; duration=0sec 2023-03-20 10:58:38,681 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:58:38,681 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 10:58:38,682 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37716 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 10:58:38,682 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): 00ef1cabe628ad083ccaa287ccbbfb01/C is initiating minor compaction (all files) 2023-03-20 10:58:38,683 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 00ef1cabe628ad083ccaa287ccbbfb01/C in TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:38,683 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/326b51c674924e818dcc27ae1ccde1a9, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/b0a463f587544d0c8bbc11c51bb06f29, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/364f078934d743daba9057b04ef90d70] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp, totalSize=36.8 K 2023-03-20 10:58:38,683 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 326b51c674924e818dcc27ae1ccde1a9, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=554, earliestPutTs=1679309915399 2023-03-20 10:58:38,683 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting b0a463f587544d0c8bbc11c51bb06f29, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=670, earliestPutTs=1679309916057 2023-03-20 10:58:38,683 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 00ef1cabe628ad083ccaa287ccbbfb01/A of 00ef1cabe628ad083ccaa287ccbbfb01 into bb99d7201d054b5b971434e1afc853f2(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:58:38,683 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 00ef1cabe628ad083ccaa287ccbbfb01: 2023-03-20 10:58:38,683 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 364f078934d743daba9057b04ef90d70, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=690, earliestPutTs=1679309916106 2023-03-20 10:58:38,684 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01., storeName=00ef1cabe628ad083ccaa287ccbbfb01/A, priority=13, startTime=1679309918263; duration=0sec 2023-03-20 10:58:38,684 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:58:38,688 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): 00ef1cabe628ad083ccaa287ccbbfb01#C#compaction#498 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:58:39,096 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/C/98b3349030e94598813de789fe7e1e1c as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/98b3349030e94598813de789fe7e1e1c 2023-03-20 10:58:39,100 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 00ef1cabe628ad083ccaa287ccbbfb01/C of 00ef1cabe628ad083ccaa287ccbbfb01 into 98b3349030e94598813de789fe7e1e1c(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:58:39,100 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 00ef1cabe628ad083ccaa287ccbbfb01: 2023-03-20 10:58:39,100 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01., storeName=00ef1cabe628ad083ccaa287ccbbfb01/C, priority=13, startTime=1679309918263; duration=0sec 2023-03-20 10:58:39,100 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:58:46,303 DEBUG [Thread-2963] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2a5d7fcf to 127.0.0.1:60226 2023-03-20 10:58:46,303 DEBUG [Thread-2963] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 10:58:46,303 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-20 10:58:46,303 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(394): wrote 126 2023-03-20 10:58:46,304 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(394): wrote 164 2023-03-20 10:58:46,304 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(394): wrote 143 2023-03-20 10:58:46,304 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(394): wrote 129 2023-03-20 10:58:46,304 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(394): wrote 14 2023-03-20 10:58:46,304 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-20 10:58:46,304 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-20 10:58:46,304 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(402): scanned 1454 2023-03-20 10:58:46,304 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(403): verified 4362 rows 2023-03-20 10:58:46,304 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(402): scanned 1467 2023-03-20 10:58:46,304 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(403): verified 4401 rows 2023-03-20 10:58:46,304 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(402): scanned 1465 2023-03-20 10:58:46,305 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(403): verified 4395 rows 2023-03-20 10:58:46,305 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(402): scanned 1486 2023-03-20 10:58:46,305 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(403): verified 4458 rows 2023-03-20 10:58:46,305 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(402): scanned 1473 2023-03-20 10:58:46,305 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(403): verified 4419 rows 2023-03-20 10:58:46,305 INFO [Listener at localhost.localdomain/43509] client.ConnectionImplementation(1959): Closing master protocol: MasterService 2023-03-20 10:58:46,305 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1a885842 to 127.0.0.1:60226 2023-03-20 10:58:46,305 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 10:58:46,310 INFO [Listener at localhost.localdomain/43509] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-20 10:58:46,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-03-20 10:58:46,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure2.ProcedureExecutor(1029): Stored pid=50, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-20 10:58:46,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-20 10:58:46,314 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679309926314"}]},"ts":"1679309926314"} 2023-03-20 10:58:46,315 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-20 10:58:46,319 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-20 10:58:46,320 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=00ef1cabe628ad083ccaa287ccbbfb01, UNASSIGN}] 2023-03-20 10:58:46,321 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=00ef1cabe628ad083ccaa287ccbbfb01, UNASSIGN 2023-03-20 10:58:46,321 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=00ef1cabe628ad083ccaa287ccbbfb01, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:46,321 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679309926321"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679309926321"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679309926321"}]},"ts":"1679309926321"} 2023-03-20 10:58:46,322 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=52, ppid=51, state=RUNNABLE; CloseRegionProcedure 00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062}] 2023-03-20 10:58:46,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-20 10:58:46,473 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 00ef1cabe628ad083ccaa287ccbbfb01 2023-03-20 10:58:46,473 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 00ef1cabe628ad083ccaa287ccbbfb01, disabling compactions & flushes 2023-03-20 10:58:46,473 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:46,473 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:46,473 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. after waiting 0 ms 2023-03-20 10:58:46,473 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:46,473 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 00ef1cabe628ad083ccaa287ccbbfb01 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-20 10:58:46,474 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=A 2023-03-20 10:58:46,474 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:46,474 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=B 2023-03-20 10:58:46,474 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:46,474 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 00ef1cabe628ad083ccaa287ccbbfb01, store=C 2023-03-20 10:58:46,474 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:46,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-20 10:58:46,880 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=700 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/A/ca5ce8e03276484f85010738d9da331c 2023-03-20 10:58:46,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-20 10:58:47,292 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=700 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/B/07a5c590a97d417b8b51da71baac2296 2023-03-20 10:58:47,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-20 10:58:47,703 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=700 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/C/22a7bf4e1eaf44389792db1012bf85fc 2023-03-20 10:58:47,716 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/A/ca5ce8e03276484f85010738d9da331c as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/ca5ce8e03276484f85010738d9da331c 2023-03-20 10:58:47,720 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/ca5ce8e03276484f85010738d9da331c, entries=100, sequenceid=700, filesize=9.5 K 2023-03-20 10:58:47,721 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/B/07a5c590a97d417b8b51da71baac2296 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/07a5c590a97d417b8b51da71baac2296 2023-03-20 10:58:47,726 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/07a5c590a97d417b8b51da71baac2296, entries=100, sequenceid=700, filesize=9.5 K 2023-03-20 10:58:47,726 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/.tmp/C/22a7bf4e1eaf44389792db1012bf85fc as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/22a7bf4e1eaf44389792db1012bf85fc 2023-03-20 10:58:47,731 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/22a7bf4e1eaf44389792db1012bf85fc, entries=100, sequenceid=700, filesize=9.5 K 2023-03-20 10:58:47,732 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 00ef1cabe628ad083ccaa287ccbbfb01 in 1259ms, sequenceid=700, compaction requested=false 2023-03-20 10:58:47,756 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/e7708c54b7694de68311caada8bc0732, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/b7644e97e9f44f5d816e140bbe3dafca, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/109b7c544e0f4d6baf81fd6b5d6b8a9c, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/f901a2b38200433789bca7579863abef, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/236e21770d6445d6afdf1065d7372524, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/b6592faf46c94b4e92aa4f9ae553140f, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/e69abef3b8024b1a9ae3b3ae61d4ac46, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/4388784eb4b0421f970250da1e392842, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/6f8722ef187e4cd8ae90dab405235d4b, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/3ef59604ecbf4790a08011bd978a5ccc, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/d1f0f3e8968a4e61bd90ccec27b410b7, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/ee711ea0dc81455bb7a494f36fd5d371, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/344e9944b1bc4a399c934b16eac07621, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/e2950c7a878747c38c81394aaf5cc938, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/c205c353a77f421e95819ebc08952346, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/974a9baf2d114c9aa5d90428dd5c5b13, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/af065d914f91473a913141b8f5953e25, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/0d8aa504581d4044b07483e2f5d04833, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/1a8957c9c7d64a03ac40bcb322a0a01e, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/c4d9fa5e692c49029c913c67f37b0ce1, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/efde1a1efeee43fb9926b2c0ad320083, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/194e7bf2c6c54d53887b2a5242c308a0, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/00018e346427427693d9eb2b89b8ad17, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/8c7e0e64f9c44d2d8cfd34ecdb68107b, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/5612ae12ced0485588c8494c33d0960b, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/2a287692814448bd8632eb503a9b93fa, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/ef4d97a8f7cc4832b30bf2fce5e93b05, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/09f1d94a719242d6a6ec232a13252f10, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/d13857e2b8e24461a2ac292d89aa7bc2, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/0a900540558c4a53a06cd7238db3fb57, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/56b61faa6bec4e9c8c49a2794744e0dc, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/75c2687799ff4c0282f76e2289308cca, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/c293b7859f4249859903025e24196204, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/79cc21c1fa634956ad9dc8e87b27e480, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/a2f1cb0f51264a0696fa9d2be1a93725, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/536cf75feb5243998bb57de793a3eb72, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/72ac9f7690124fd6a817a0becdb222a1, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/f580d93e4e36496fac178d72fa0fdb38] to archive 2023-03-20 10:58:47,757 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-20 10:58:47,758 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/e7708c54b7694de68311caada8bc0732 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/e7708c54b7694de68311caada8bc0732 2023-03-20 10:58:47,759 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/b7644e97e9f44f5d816e140bbe3dafca to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/b7644e97e9f44f5d816e140bbe3dafca 2023-03-20 10:58:47,760 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/109b7c544e0f4d6baf81fd6b5d6b8a9c to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/109b7c544e0f4d6baf81fd6b5d6b8a9c 2023-03-20 10:58:47,761 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/f901a2b38200433789bca7579863abef to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/f901a2b38200433789bca7579863abef 2023-03-20 10:58:47,761 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/236e21770d6445d6afdf1065d7372524 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/236e21770d6445d6afdf1065d7372524 2023-03-20 10:58:47,762 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/b6592faf46c94b4e92aa4f9ae553140f to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/b6592faf46c94b4e92aa4f9ae553140f 2023-03-20 10:58:47,763 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/e69abef3b8024b1a9ae3b3ae61d4ac46 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/e69abef3b8024b1a9ae3b3ae61d4ac46 2023-03-20 10:58:47,764 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/4388784eb4b0421f970250da1e392842 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/4388784eb4b0421f970250da1e392842 2023-03-20 10:58:47,765 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/6f8722ef187e4cd8ae90dab405235d4b to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/6f8722ef187e4cd8ae90dab405235d4b 2023-03-20 10:58:47,766 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/3ef59604ecbf4790a08011bd978a5ccc to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/3ef59604ecbf4790a08011bd978a5ccc 2023-03-20 10:58:47,767 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/d1f0f3e8968a4e61bd90ccec27b410b7 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/d1f0f3e8968a4e61bd90ccec27b410b7 2023-03-20 10:58:47,768 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/ee711ea0dc81455bb7a494f36fd5d371 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/ee711ea0dc81455bb7a494f36fd5d371 2023-03-20 10:58:47,769 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/344e9944b1bc4a399c934b16eac07621 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/344e9944b1bc4a399c934b16eac07621 2023-03-20 10:58:47,770 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/e2950c7a878747c38c81394aaf5cc938 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/e2950c7a878747c38c81394aaf5cc938 2023-03-20 10:58:47,771 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/c205c353a77f421e95819ebc08952346 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/c205c353a77f421e95819ebc08952346 2023-03-20 10:58:47,772 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/974a9baf2d114c9aa5d90428dd5c5b13 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/974a9baf2d114c9aa5d90428dd5c5b13 2023-03-20 10:58:47,773 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/af065d914f91473a913141b8f5953e25 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/af065d914f91473a913141b8f5953e25 2023-03-20 10:58:47,774 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/0d8aa504581d4044b07483e2f5d04833 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/0d8aa504581d4044b07483e2f5d04833 2023-03-20 10:58:47,775 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/1a8957c9c7d64a03ac40bcb322a0a01e to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/1a8957c9c7d64a03ac40bcb322a0a01e 2023-03-20 10:58:47,776 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/c4d9fa5e692c49029c913c67f37b0ce1 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/c4d9fa5e692c49029c913c67f37b0ce1 2023-03-20 10:58:47,777 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/efde1a1efeee43fb9926b2c0ad320083 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/efde1a1efeee43fb9926b2c0ad320083 2023-03-20 10:58:47,778 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/194e7bf2c6c54d53887b2a5242c308a0 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/194e7bf2c6c54d53887b2a5242c308a0 2023-03-20 10:58:47,779 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/00018e346427427693d9eb2b89b8ad17 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/00018e346427427693d9eb2b89b8ad17 2023-03-20 10:58:47,779 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/8c7e0e64f9c44d2d8cfd34ecdb68107b to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/8c7e0e64f9c44d2d8cfd34ecdb68107b 2023-03-20 10:58:47,780 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/5612ae12ced0485588c8494c33d0960b to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/5612ae12ced0485588c8494c33d0960b 2023-03-20 10:58:47,781 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/2a287692814448bd8632eb503a9b93fa to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/2a287692814448bd8632eb503a9b93fa 2023-03-20 10:58:47,782 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/ef4d97a8f7cc4832b30bf2fce5e93b05 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/ef4d97a8f7cc4832b30bf2fce5e93b05 2023-03-20 10:58:47,783 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/09f1d94a719242d6a6ec232a13252f10 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/09f1d94a719242d6a6ec232a13252f10 2023-03-20 10:58:47,784 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/d13857e2b8e24461a2ac292d89aa7bc2 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/d13857e2b8e24461a2ac292d89aa7bc2 2023-03-20 10:58:47,785 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/0a900540558c4a53a06cd7238db3fb57 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/0a900540558c4a53a06cd7238db3fb57 2023-03-20 10:58:47,786 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/56b61faa6bec4e9c8c49a2794744e0dc to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/56b61faa6bec4e9c8c49a2794744e0dc 2023-03-20 10:58:47,788 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/75c2687799ff4c0282f76e2289308cca to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/75c2687799ff4c0282f76e2289308cca 2023-03-20 10:58:47,789 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/c293b7859f4249859903025e24196204 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/c293b7859f4249859903025e24196204 2023-03-20 10:58:47,790 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/79cc21c1fa634956ad9dc8e87b27e480 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/79cc21c1fa634956ad9dc8e87b27e480 2023-03-20 10:58:47,791 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/a2f1cb0f51264a0696fa9d2be1a93725 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/a2f1cb0f51264a0696fa9d2be1a93725 2023-03-20 10:58:47,792 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/536cf75feb5243998bb57de793a3eb72 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/536cf75feb5243998bb57de793a3eb72 2023-03-20 10:58:47,793 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/72ac9f7690124fd6a817a0becdb222a1 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/72ac9f7690124fd6a817a0becdb222a1 2023-03-20 10:58:47,794 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/f580d93e4e36496fac178d72fa0fdb38 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/f580d93e4e36496fac178d72fa0fdb38 2023-03-20 10:58:47,799 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/e73d1f22ec8d48ef99ee60c3ad764200, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/f92efa50737c44c3842a479aefb28315, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/715a01a8cd4f4b45bcb8fb5e74535872, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/788d0d925cb0412d8c2ec20e8ca2b149, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/fb5e64fd6a244eaba8f99f120d121c1b, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/4baa322043ce48a6979a83563875bf6d, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/e6f5408f90e34443b17c0cb00c201ee2, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/bbf408bb9414416d922cb9b4cfcca676, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/f82a203e14e94decb2bc0ee06b14cacb, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/7594663222dd445db928fd873cf75b47, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/ee5ae351142243ff83418d436234a9d8, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/0d4cb271aaf54ce1b924a495aa98f312, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/8c8906e63959467ca2abd8aafb6abe99, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/4356b31834ab4ea99e4c06f88b9c8a31, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/c254b8df9d724d40985885924460caee, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/8f08871fecc5470097849a1397920f86, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/e2d1d7ce6a8f4af99314b1d8d1f4f6e2, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/3a85d29595c24ac49b934b5d73e5edfb, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/b23380a9299748aaa5c6c69924359972, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/e2ae5ec9a3a94bb086898d91885f375e, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/50a474adf301439fb58dd9cb295ae24e, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/8c3417ca609347ec9f19eb82eafb405f, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/4f098ddb87f2443696c67bb63dbcc827, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/6e4e66df9a394e28aa66f0ebb6c22a25, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/e40f73d32d324ab8acdc8fef5849e93c, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/3d73971d639447a3a78d7d6ebb77b884, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/d077d587b1004f5bbfa31131d33d67a0, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/caa40d24be7e40118a1e736f96a63e09, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/b3b3b8fdeae24f778c5d4d8559cbf279, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/3983ea79ff1c4636818fce7330fccbfd, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/18861f4e90f8416a801e645786b8fab0, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/c3ac0df799f14e8197beaf4a00bceb2a, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/523a8a7b45e244d8986be99111857c61, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/68adbb5b4f9f4db0b19bcbf2155995bf, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/fb375bafdae141db9c515ba5e7013234, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/c376fa9bc3c94c088cd5b6d14cdfa700, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/a9b1f1a1ba664ae885192b94e2b71e49, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/1af1575a18554efcbdb6a1828ffe8f79] to archive 2023-03-20 10:58:47,799 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-20 10:58:47,801 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/e73d1f22ec8d48ef99ee60c3ad764200 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/e73d1f22ec8d48ef99ee60c3ad764200 2023-03-20 10:58:47,802 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/f92efa50737c44c3842a479aefb28315 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/f92efa50737c44c3842a479aefb28315 2023-03-20 10:58:47,803 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/715a01a8cd4f4b45bcb8fb5e74535872 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/715a01a8cd4f4b45bcb8fb5e74535872 2023-03-20 10:58:47,803 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/788d0d925cb0412d8c2ec20e8ca2b149 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/788d0d925cb0412d8c2ec20e8ca2b149 2023-03-20 10:58:47,804 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/fb5e64fd6a244eaba8f99f120d121c1b to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/fb5e64fd6a244eaba8f99f120d121c1b 2023-03-20 10:58:47,805 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/4baa322043ce48a6979a83563875bf6d to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/4baa322043ce48a6979a83563875bf6d 2023-03-20 10:58:47,806 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/e6f5408f90e34443b17c0cb00c201ee2 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/e6f5408f90e34443b17c0cb00c201ee2 2023-03-20 10:58:47,807 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/bbf408bb9414416d922cb9b4cfcca676 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/bbf408bb9414416d922cb9b4cfcca676 2023-03-20 10:58:47,808 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/f82a203e14e94decb2bc0ee06b14cacb to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/f82a203e14e94decb2bc0ee06b14cacb 2023-03-20 10:58:47,809 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/7594663222dd445db928fd873cf75b47 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/7594663222dd445db928fd873cf75b47 2023-03-20 10:58:47,810 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/ee5ae351142243ff83418d436234a9d8 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/ee5ae351142243ff83418d436234a9d8 2023-03-20 10:58:47,811 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/0d4cb271aaf54ce1b924a495aa98f312 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/0d4cb271aaf54ce1b924a495aa98f312 2023-03-20 10:58:47,812 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/8c8906e63959467ca2abd8aafb6abe99 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/8c8906e63959467ca2abd8aafb6abe99 2023-03-20 10:58:47,813 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/4356b31834ab4ea99e4c06f88b9c8a31 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/4356b31834ab4ea99e4c06f88b9c8a31 2023-03-20 10:58:47,813 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/c254b8df9d724d40985885924460caee to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/c254b8df9d724d40985885924460caee 2023-03-20 10:58:47,814 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/8f08871fecc5470097849a1397920f86 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/8f08871fecc5470097849a1397920f86 2023-03-20 10:58:47,817 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/e2d1d7ce6a8f4af99314b1d8d1f4f6e2 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/e2d1d7ce6a8f4af99314b1d8d1f4f6e2 2023-03-20 10:58:47,819 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/3a85d29595c24ac49b934b5d73e5edfb to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/3a85d29595c24ac49b934b5d73e5edfb 2023-03-20 10:58:47,826 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/b23380a9299748aaa5c6c69924359972 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/b23380a9299748aaa5c6c69924359972 2023-03-20 10:58:47,828 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/e2ae5ec9a3a94bb086898d91885f375e to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/e2ae5ec9a3a94bb086898d91885f375e 2023-03-20 10:58:47,829 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/50a474adf301439fb58dd9cb295ae24e to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/50a474adf301439fb58dd9cb295ae24e 2023-03-20 10:58:47,830 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/8c3417ca609347ec9f19eb82eafb405f to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/8c3417ca609347ec9f19eb82eafb405f 2023-03-20 10:58:47,831 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/4f098ddb87f2443696c67bb63dbcc827 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/4f098ddb87f2443696c67bb63dbcc827 2023-03-20 10:58:47,832 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/6e4e66df9a394e28aa66f0ebb6c22a25 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/6e4e66df9a394e28aa66f0ebb6c22a25 2023-03-20 10:58:47,833 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/e40f73d32d324ab8acdc8fef5849e93c to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/e40f73d32d324ab8acdc8fef5849e93c 2023-03-20 10:58:47,834 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/3d73971d639447a3a78d7d6ebb77b884 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/3d73971d639447a3a78d7d6ebb77b884 2023-03-20 10:58:47,835 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/d077d587b1004f5bbfa31131d33d67a0 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/d077d587b1004f5bbfa31131d33d67a0 2023-03-20 10:58:47,836 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/caa40d24be7e40118a1e736f96a63e09 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/caa40d24be7e40118a1e736f96a63e09 2023-03-20 10:58:47,837 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/b3b3b8fdeae24f778c5d4d8559cbf279 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/b3b3b8fdeae24f778c5d4d8559cbf279 2023-03-20 10:58:47,838 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/3983ea79ff1c4636818fce7330fccbfd to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/3983ea79ff1c4636818fce7330fccbfd 2023-03-20 10:58:47,839 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/18861f4e90f8416a801e645786b8fab0 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/18861f4e90f8416a801e645786b8fab0 2023-03-20 10:58:47,841 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/c3ac0df799f14e8197beaf4a00bceb2a to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/c3ac0df799f14e8197beaf4a00bceb2a 2023-03-20 10:58:47,843 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/523a8a7b45e244d8986be99111857c61 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/523a8a7b45e244d8986be99111857c61 2023-03-20 10:58:47,844 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/68adbb5b4f9f4db0b19bcbf2155995bf to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/68adbb5b4f9f4db0b19bcbf2155995bf 2023-03-20 10:58:47,844 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/fb375bafdae141db9c515ba5e7013234 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/fb375bafdae141db9c515ba5e7013234 2023-03-20 10:58:47,845 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/c376fa9bc3c94c088cd5b6d14cdfa700 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/c376fa9bc3c94c088cd5b6d14cdfa700 2023-03-20 10:58:47,846 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/a9b1f1a1ba664ae885192b94e2b71e49 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/a9b1f1a1ba664ae885192b94e2b71e49 2023-03-20 10:58:47,847 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/1af1575a18554efcbdb6a1828ffe8f79 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/1af1575a18554efcbdb6a1828ffe8f79 2023-03-20 10:58:47,850 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/4cd7075d7aff49879958571076203591, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/c4522be77b084496852c128262f434d9, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/ca97ba0489744cfe8516c598d74ddddc, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/c63dc7ef0cf14aeb999cbc9bf5f53846, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/04b8974f07564d7a8c6c9150497ab5a4, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/bc20d1eeffd3454388980049639db488, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/916e137f601944b5a5930ad568f2aecb, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/21dd6e07cfe14bf4a78ac7e660acb83f, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/c0be06edbe84449fb269f3e4e3c2bb88, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/fd82ec6f3dc74f609ce51093b779cd32, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/14937910ff8540c3b1bedc30281a2b37, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/81a11da5ee5e4871800279b4514864cf, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/ce7e68b49a854fb5848a591dae576208, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/3471aa96965a46b7aa43ac1b7ac2d4b7, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/c6617e014ddf4699a5629c6f2e59b088, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/e189b48dcee64e089821662663d83bc7, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/1ab635d4435647eda5ef950d212c8dcf, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/7a6f551878f844a5b1260f726d2fd5d1, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/744d16a4163f4b51b79fff199742c01b, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/d60e7f3f796e41e1ab229e8c1f483a9d, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/4ad090d16856438e9743f3d3df12656b, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/dcb2407086cf4407abb1c3c951e8f2c4, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/3035d4c7048d43d1805a20c6bec51161, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/b5429c5acff849419ef3f9d176c90022, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/78a01a089e6249d29ef2c7dac3d2f164, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/c41e0c6f991343e4963598edea8cf3d8, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/cc2521458e4c45db817d9e61eabd2d0e, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/2a1568ef043749d68091bf55f254c32e, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/665612b9488a4c8f901ec084028c3314, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/b292772694454a79a0d284bec3be81d4, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/e3ae9764d49f413391052fe25f16823b, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/c846580377ac412c8fd5c746716be020, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/0426c1c54e414f00bdc8d8db5770348e, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/326b51c674924e818dcc27ae1ccde1a9, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/979d99d0f85b4090bb3c064e5eca9c0f, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/b0a463f587544d0c8bbc11c51bb06f29, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/364f078934d743daba9057b04ef90d70] to archive 2023-03-20 10:58:47,851 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-20 10:58:47,852 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/4cd7075d7aff49879958571076203591 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/4cd7075d7aff49879958571076203591 2023-03-20 10:58:47,853 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/c4522be77b084496852c128262f434d9 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/c4522be77b084496852c128262f434d9 2023-03-20 10:58:47,854 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/ca97ba0489744cfe8516c598d74ddddc to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/ca97ba0489744cfe8516c598d74ddddc 2023-03-20 10:58:47,855 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/c63dc7ef0cf14aeb999cbc9bf5f53846 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/c63dc7ef0cf14aeb999cbc9bf5f53846 2023-03-20 10:58:47,858 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/04b8974f07564d7a8c6c9150497ab5a4 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/04b8974f07564d7a8c6c9150497ab5a4 2023-03-20 10:58:47,860 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/bc20d1eeffd3454388980049639db488 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/bc20d1eeffd3454388980049639db488 2023-03-20 10:58:47,861 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/916e137f601944b5a5930ad568f2aecb to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/916e137f601944b5a5930ad568f2aecb 2023-03-20 10:58:47,862 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/21dd6e07cfe14bf4a78ac7e660acb83f to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/21dd6e07cfe14bf4a78ac7e660acb83f 2023-03-20 10:58:47,863 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/c0be06edbe84449fb269f3e4e3c2bb88 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/c0be06edbe84449fb269f3e4e3c2bb88 2023-03-20 10:58:47,864 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/fd82ec6f3dc74f609ce51093b779cd32 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/fd82ec6f3dc74f609ce51093b779cd32 2023-03-20 10:58:47,865 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/14937910ff8540c3b1bedc30281a2b37 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/14937910ff8540c3b1bedc30281a2b37 2023-03-20 10:58:47,866 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/81a11da5ee5e4871800279b4514864cf to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/81a11da5ee5e4871800279b4514864cf 2023-03-20 10:58:47,867 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/ce7e68b49a854fb5848a591dae576208 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/ce7e68b49a854fb5848a591dae576208 2023-03-20 10:58:47,868 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/3471aa96965a46b7aa43ac1b7ac2d4b7 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/3471aa96965a46b7aa43ac1b7ac2d4b7 2023-03-20 10:58:47,869 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/c6617e014ddf4699a5629c6f2e59b088 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/c6617e014ddf4699a5629c6f2e59b088 2023-03-20 10:58:47,870 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/e189b48dcee64e089821662663d83bc7 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/e189b48dcee64e089821662663d83bc7 2023-03-20 10:58:47,871 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/1ab635d4435647eda5ef950d212c8dcf to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/1ab635d4435647eda5ef950d212c8dcf 2023-03-20 10:58:47,871 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/7a6f551878f844a5b1260f726d2fd5d1 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/7a6f551878f844a5b1260f726d2fd5d1 2023-03-20 10:58:47,872 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/744d16a4163f4b51b79fff199742c01b to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/744d16a4163f4b51b79fff199742c01b 2023-03-20 10:58:47,873 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/d60e7f3f796e41e1ab229e8c1f483a9d to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/d60e7f3f796e41e1ab229e8c1f483a9d 2023-03-20 10:58:47,873 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/4ad090d16856438e9743f3d3df12656b to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/4ad090d16856438e9743f3d3df12656b 2023-03-20 10:58:47,874 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/dcb2407086cf4407abb1c3c951e8f2c4 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/dcb2407086cf4407abb1c3c951e8f2c4 2023-03-20 10:58:47,874 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/3035d4c7048d43d1805a20c6bec51161 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/3035d4c7048d43d1805a20c6bec51161 2023-03-20 10:58:47,875 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/b5429c5acff849419ef3f9d176c90022 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/b5429c5acff849419ef3f9d176c90022 2023-03-20 10:58:47,876 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/78a01a089e6249d29ef2c7dac3d2f164 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/78a01a089e6249d29ef2c7dac3d2f164 2023-03-20 10:58:47,876 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/c41e0c6f991343e4963598edea8cf3d8 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/c41e0c6f991343e4963598edea8cf3d8 2023-03-20 10:58:47,877 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/cc2521458e4c45db817d9e61eabd2d0e to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/cc2521458e4c45db817d9e61eabd2d0e 2023-03-20 10:58:47,878 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/2a1568ef043749d68091bf55f254c32e to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/2a1568ef043749d68091bf55f254c32e 2023-03-20 10:58:47,879 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/665612b9488a4c8f901ec084028c3314 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/665612b9488a4c8f901ec084028c3314 2023-03-20 10:58:47,880 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/b292772694454a79a0d284bec3be81d4 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/b292772694454a79a0d284bec3be81d4 2023-03-20 10:58:47,881 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/e3ae9764d49f413391052fe25f16823b to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/e3ae9764d49f413391052fe25f16823b 2023-03-20 10:58:47,881 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/c846580377ac412c8fd5c746716be020 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/c846580377ac412c8fd5c746716be020 2023-03-20 10:58:47,882 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/0426c1c54e414f00bdc8d8db5770348e to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/0426c1c54e414f00bdc8d8db5770348e 2023-03-20 10:58:47,883 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/326b51c674924e818dcc27ae1ccde1a9 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/326b51c674924e818dcc27ae1ccde1a9 2023-03-20 10:58:47,884 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/979d99d0f85b4090bb3c064e5eca9c0f to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/979d99d0f85b4090bb3c064e5eca9c0f 2023-03-20 10:58:47,885 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/b0a463f587544d0c8bbc11c51bb06f29 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/b0a463f587544d0c8bbc11c51bb06f29 2023-03-20 10:58:47,885 DEBUG [StoreCloser-TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/364f078934d743daba9057b04ef90d70 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/364f078934d743daba9057b04ef90d70 2023-03-20 10:58:47,890 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/recovered.edits/703.seqid, newMaxSeqId=703, maxSeqId=1 2023-03-20 10:58:47,892 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01. 2023-03-20 10:58:47,892 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 00ef1cabe628ad083ccaa287ccbbfb01: 2023-03-20 10:58:47,894 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 00ef1cabe628ad083ccaa287ccbbfb01 2023-03-20 10:58:47,894 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=00ef1cabe628ad083ccaa287ccbbfb01, regionState=CLOSED 2023-03-20 10:58:47,894 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679309927894"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679309927894"}]},"ts":"1679309927894"} 2023-03-20 10:58:47,897 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=52, resume processing ppid=51 2023-03-20 10:58:47,897 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=51, state=SUCCESS; CloseRegionProcedure 00ef1cabe628ad083ccaa287ccbbfb01, server=jenkins-hbase20.apache.org,36041,1679309777062 in 1.5740 sec 2023-03-20 10:58:47,899 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=50 2023-03-20 10:58:47,899 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=50, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=00ef1cabe628ad083ccaa287ccbbfb01, UNASSIGN in 1.5780 sec 2023-03-20 10:58:47,899 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679309927899"}]},"ts":"1679309927899"} 2023-03-20 10:58:47,900 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-20 10:58:47,902 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-20 10:58:47,904 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=50, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5910 sec 2023-03-20 10:58:48,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-20 10:58:48,418 INFO [Listener at localhost.localdomain/43509] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 50 completed 2023-03-20 10:58:48,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-03-20 10:58:48,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure2.ProcedureExecutor(1029): Stored pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-20 10:58:48,421 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-20 10:58:48,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-03-20 10:58:48,422 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=53, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-20 10:58:48,429 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01 2023-03-20 10:58:48,430 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A, FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B, FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C, FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/recovered.edits] 2023-03-20 10:58:48,433 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/bb99d7201d054b5b971434e1afc853f2 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/bb99d7201d054b5b971434e1afc853f2 2023-03-20 10:58:48,434 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/ca5ce8e03276484f85010738d9da331c to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/A/ca5ce8e03276484f85010738d9da331c 2023-03-20 10:58:48,436 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/07a5c590a97d417b8b51da71baac2296 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/07a5c590a97d417b8b51da71baac2296 2023-03-20 10:58:48,436 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/bb17460e81db421881c7e9c97b0c919f to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/B/bb17460e81db421881c7e9c97b0c919f 2023-03-20 10:58:48,438 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/22a7bf4e1eaf44389792db1012bf85fc to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/22a7bf4e1eaf44389792db1012bf85fc 2023-03-20 10:58:48,439 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/98b3349030e94598813de789fe7e1e1c to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/C/98b3349030e94598813de789fe7e1e1c 2023-03-20 10:58:48,441 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/recovered.edits/703.seqid to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01/recovered.edits/703.seqid 2023-03-20 10:58:48,442 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/00ef1cabe628ad083ccaa287ccbbfb01 2023-03-20 10:58:48,442 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-20 10:58:48,443 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=53, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-20 10:58:48,445 WARN [PEWorker-4] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-20 10:58:48,446 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-20 10:58:48,447 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=53, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-20 10:58:48,447 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-20 10:58:48,448 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1679309928448"}]},"ts":"9223372036854775807"} 2023-03-20 10:58:48,450 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-20 10:58:48,450 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 00ef1cabe628ad083ccaa287ccbbfb01, NAME => 'TestAcidGuarantees,,1679309895489.00ef1cabe628ad083ccaa287ccbbfb01.', STARTKEY => '', ENDKEY => ''}] 2023-03-20 10:58:48,450 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-20 10:58:48,450 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1679309928450"}]},"ts":"9223372036854775807"} 2023-03-20 10:58:48,454 INFO [PEWorker-4] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-20 10:58:48,455 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(130): Finished pid=53, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-20 10:58:48,456 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=53, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 36 msec 2023-03-20 10:58:48,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-03-20 10:58:48,523 INFO [Listener at localhost.localdomain/43509] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 53 completed 2023-03-20 10:58:48,538 INFO [Listener at localhost.localdomain/43509] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=225 (was 227), OpenFileDescriptor=551 (was 553), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=717 (was 775), ProcessCount=184 (was 184), AvailableMemoryMB=9056 (was 8954) - AvailableMemoryMB LEAK? - 2023-03-20 10:58:48,555 INFO [Listener at localhost.localdomain/43509] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=225, OpenFileDescriptor=551, MaxFileDescriptor=60000, SystemLoadAverage=717, ProcessCount=184, AvailableMemoryMB=9054 2023-03-20 10:58:48,556 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-20 10:58:48,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.HMaster$4(2112): Client=jenkins//148.251.75.209 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-20 10:58:48,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure2.ProcedureExecutor(1029): Stored pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-20 10:58:48,560 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-20 10:58:48,560 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(697): Client=jenkins//148.251.75.209 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 54 2023-03-20 10:58:48,560 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-20 10:58:48,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-20 10:58:48,561 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5 2023-03-20 10:58:48,562 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5 empty. 2023-03-20 10:58:48,562 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5 2023-03-20 10:58:48,562 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-20 10:58:48,573 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-20 10:58:48,574 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 96c44321a6ff3c73883afe25fc999ea5, NAME => 'TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp 2023-03-20 10:58:48,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-20 10:58:48,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-20 10:58:48,984 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-20 10:58:48,984 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 96c44321a6ff3c73883afe25fc999ea5, disabling compactions & flushes 2023-03-20 10:58:48,984 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:48,984 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:48,984 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. after waiting 0 ms 2023-03-20 10:58:48,984 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:48,984 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:48,984 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 96c44321a6ff3c73883afe25fc999ea5: 2023-03-20 10:58:48,987 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-20 10:58:48,988 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679309928988"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679309928988"}]},"ts":"1679309928988"} 2023-03-20 10:58:48,989 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-20 10:58:48,990 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-20 10:58:48,990 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679309928990"}]},"ts":"1679309928990"} 2023-03-20 10:58:48,991 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-20 10:58:48,993 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=96c44321a6ff3c73883afe25fc999ea5, ASSIGN}] 2023-03-20 10:58:48,995 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=96c44321a6ff3c73883afe25fc999ea5, ASSIGN 2023-03-20 10:58:48,996 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=96c44321a6ff3c73883afe25fc999ea5, ASSIGN; state=OFFLINE, location=jenkins-hbase20.apache.org,36041,1679309777062; forceNewPlan=false, retain=false 2023-03-20 10:58:49,147 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=96c44321a6ff3c73883afe25fc999ea5, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:49,147 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679309929147"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679309929147"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679309929147"}]},"ts":"1679309929147"} 2023-03-20 10:58:49,149 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=55, state=RUNNABLE; OpenRegionProcedure 96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062}] 2023-03-20 10:58:49,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-20 10:58:49,305 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:49,305 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 96c44321a6ff3c73883afe25fc999ea5, NAME => 'TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.', STARTKEY => '', ENDKEY => ''} 2023-03-20 10:58:49,306 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 96c44321a6ff3c73883afe25fc999ea5 2023-03-20 10:58:49,306 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-20 10:58:49,306 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 96c44321a6ff3c73883afe25fc999ea5 2023-03-20 10:58:49,306 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 96c44321a6ff3c73883afe25fc999ea5 2023-03-20 10:58:49,308 INFO [StoreOpener-96c44321a6ff3c73883afe25fc999ea5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 96c44321a6ff3c73883afe25fc999ea5 2023-03-20 10:58:49,311 DEBUG [StoreOpener-96c44321a6ff3c73883afe25fc999ea5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A 2023-03-20 10:58:49,311 DEBUG [StoreOpener-96c44321a6ff3c73883afe25fc999ea5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A 2023-03-20 10:58:49,312 INFO [StoreOpener-96c44321a6ff3c73883afe25fc999ea5-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-20 10:58:49,312 INFO [StoreOpener-96c44321a6ff3c73883afe25fc999ea5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 96c44321a6ff3c73883afe25fc999ea5 columnFamilyName A 2023-03-20 10:58:49,313 INFO [StoreOpener-96c44321a6ff3c73883afe25fc999ea5-1] regionserver.HStore(310): Store=96c44321a6ff3c73883afe25fc999ea5/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-20 10:58:49,313 INFO [StoreOpener-96c44321a6ff3c73883afe25fc999ea5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 96c44321a6ff3c73883afe25fc999ea5 2023-03-20 10:58:49,316 DEBUG [StoreOpener-96c44321a6ff3c73883afe25fc999ea5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B 2023-03-20 10:58:49,316 DEBUG [StoreOpener-96c44321a6ff3c73883afe25fc999ea5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B 2023-03-20 10:58:49,316 INFO [StoreOpener-96c44321a6ff3c73883afe25fc999ea5-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-20 10:58:49,316 INFO [StoreOpener-96c44321a6ff3c73883afe25fc999ea5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 96c44321a6ff3c73883afe25fc999ea5 columnFamilyName B 2023-03-20 10:58:49,317 INFO [StoreOpener-96c44321a6ff3c73883afe25fc999ea5-1] regionserver.HStore(310): Store=96c44321a6ff3c73883afe25fc999ea5/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-20 10:58:49,317 INFO [StoreOpener-96c44321a6ff3c73883afe25fc999ea5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 96c44321a6ff3c73883afe25fc999ea5 2023-03-20 10:58:49,319 DEBUG [StoreOpener-96c44321a6ff3c73883afe25fc999ea5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C 2023-03-20 10:58:49,319 DEBUG [StoreOpener-96c44321a6ff3c73883afe25fc999ea5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C 2023-03-20 10:58:49,319 INFO [StoreOpener-96c44321a6ff3c73883afe25fc999ea5-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-20 10:58:49,320 INFO [StoreOpener-96c44321a6ff3c73883afe25fc999ea5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 96c44321a6ff3c73883afe25fc999ea5 columnFamilyName C 2023-03-20 10:58:49,320 INFO [StoreOpener-96c44321a6ff3c73883afe25fc999ea5-1] regionserver.HStore(310): Store=96c44321a6ff3c73883afe25fc999ea5/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-20 10:58:49,320 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:49,321 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5 2023-03-20 10:58:49,322 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5 2023-03-20 10:58:49,325 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-20 10:58:49,326 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 96c44321a6ff3c73883afe25fc999ea5 2023-03-20 10:58:49,329 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-20 10:58:49,329 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 96c44321a6ff3c73883afe25fc999ea5; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72876698, jitterRate=0.0859474241733551}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-20 10:58:49,329 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 96c44321a6ff3c73883afe25fc999ea5: 2023-03-20 10:58:49,330 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5., pid=56, masterSystemTime=1679309929301 2023-03-20 10:58:49,333 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:49,333 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:49,334 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=96c44321a6ff3c73883afe25fc999ea5, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:49,334 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679309929334"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1679309929334"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679309929334"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679309929334"}]},"ts":"1679309929334"} 2023-03-20 10:58:49,337 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=55 2023-03-20 10:58:49,337 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=55, state=SUCCESS; OpenRegionProcedure 96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 in 186 msec 2023-03-20 10:58:49,339 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=54 2023-03-20 10:58:49,339 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=54, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=96c44321a6ff3c73883afe25fc999ea5, ASSIGN in 344 msec 2023-03-20 10:58:49,339 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-20 10:58:49,339 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679309929339"}]},"ts":"1679309929339"} 2023-03-20 10:58:49,340 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-20 10:58:49,342 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-20 10:58:49,343 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=54, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 786 msec 2023-03-20 10:58:49,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-20 10:58:49,665 INFO [Listener at localhost.localdomain/43509] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 54 completed 2023-03-20 10:58:49,665 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(139): Connect 0x4ae8f5b1 to 127.0.0.1:60226 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 10:58:49,670 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@b200ba7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 10:58:49,671 DEBUG [Listener at localhost.localdomain/43509] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 10:58:49,672 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:39782, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 10:58:49,673 DEBUG [Listener at localhost.localdomain/43509] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-20 10:58:49,676 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:36566, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-20 10:58:49,677 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-20 10:58:49,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.HMaster$12(2454): Client=jenkins//148.251.75.209 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-20 10:58:49,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure2.ProcedureExecutor(1029): Stored pid=57, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-03-20 10:58:50,090 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-20 10:58:50,091 DEBUG [PEWorker-2] util.FSTableDescriptors(519): Deleted hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-20 10:58:50,092 INFO [PEWorker-2] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-20 10:58:50,093 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-03-20 10:58:50,103 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=96c44321a6ff3c73883afe25fc999ea5, REOPEN/MOVE}] 2023-03-20 10:58:50,104 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=96c44321a6ff3c73883afe25fc999ea5, REOPEN/MOVE 2023-03-20 10:58:50,104 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=96c44321a6ff3c73883afe25fc999ea5, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:50,105 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679309930104"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679309930104"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679309930104"}]},"ts":"1679309930104"} 2023-03-20 10:58:50,106 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=59, state=RUNNABLE; CloseRegionProcedure 96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062}] 2023-03-20 10:58:50,259 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 96c44321a6ff3c73883afe25fc999ea5 2023-03-20 10:58:50,260 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 96c44321a6ff3c73883afe25fc999ea5, disabling compactions & flushes 2023-03-20 10:58:50,260 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:50,260 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:50,260 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. after waiting 0 ms 2023-03-20 10:58:50,260 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:50,264 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-03-20 10:58:50,264 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:50,264 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 96c44321a6ff3c73883afe25fc999ea5: 2023-03-20 10:58:50,264 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 96c44321a6ff3c73883afe25fc999ea5 to self. 2023-03-20 10:58:50,266 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 96c44321a6ff3c73883afe25fc999ea5 2023-03-20 10:58:50,266 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=96c44321a6ff3c73883afe25fc999ea5, regionState=CLOSED 2023-03-20 10:58:50,266 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679309930266"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679309930266"}]},"ts":"1679309930266"} 2023-03-20 10:58:50,269 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=59 2023-03-20 10:58:50,269 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=59, state=SUCCESS; CloseRegionProcedure 96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 in 161 msec 2023-03-20 10:58:50,270 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=96c44321a6ff3c73883afe25fc999ea5, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase20.apache.org,36041,1679309777062; forceNewPlan=false, retain=true 2023-03-20 10:58:50,421 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=96c44321a6ff3c73883afe25fc999ea5, regionState=OPENING, regionLocation=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:50,421 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679309930421"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679309930421"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679309930421"}]},"ts":"1679309930421"} 2023-03-20 10:58:50,423 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=59, state=RUNNABLE; OpenRegionProcedure 96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062}] 2023-03-20 10:58:50,578 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:50,578 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 96c44321a6ff3c73883afe25fc999ea5, NAME => 'TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.', STARTKEY => '', ENDKEY => ''} 2023-03-20 10:58:50,579 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 96c44321a6ff3c73883afe25fc999ea5 2023-03-20 10:58:50,579 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-20 10:58:50,579 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7894): checking encryption for 96c44321a6ff3c73883afe25fc999ea5 2023-03-20 10:58:50,579 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(7897): checking classloading for 96c44321a6ff3c73883afe25fc999ea5 2023-03-20 10:58:50,580 INFO [StoreOpener-96c44321a6ff3c73883afe25fc999ea5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 96c44321a6ff3c73883afe25fc999ea5 2023-03-20 10:58:50,581 DEBUG [StoreOpener-96c44321a6ff3c73883afe25fc999ea5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A 2023-03-20 10:58:50,581 DEBUG [StoreOpener-96c44321a6ff3c73883afe25fc999ea5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A 2023-03-20 10:58:50,581 INFO [StoreOpener-96c44321a6ff3c73883afe25fc999ea5-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-20 10:58:50,581 INFO [StoreOpener-96c44321a6ff3c73883afe25fc999ea5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 96c44321a6ff3c73883afe25fc999ea5 columnFamilyName A 2023-03-20 10:58:50,582 INFO [StoreOpener-96c44321a6ff3c73883afe25fc999ea5-1] regionserver.HStore(310): Store=96c44321a6ff3c73883afe25fc999ea5/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-20 10:58:50,583 INFO [StoreOpener-96c44321a6ff3c73883afe25fc999ea5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 96c44321a6ff3c73883afe25fc999ea5 2023-03-20 10:58:50,583 DEBUG [StoreOpener-96c44321a6ff3c73883afe25fc999ea5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B 2023-03-20 10:58:50,583 DEBUG [StoreOpener-96c44321a6ff3c73883afe25fc999ea5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B 2023-03-20 10:58:50,584 INFO [StoreOpener-96c44321a6ff3c73883afe25fc999ea5-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-20 10:58:50,584 INFO [StoreOpener-96c44321a6ff3c73883afe25fc999ea5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 96c44321a6ff3c73883afe25fc999ea5 columnFamilyName B 2023-03-20 10:58:50,584 INFO [StoreOpener-96c44321a6ff3c73883afe25fc999ea5-1] regionserver.HStore(310): Store=96c44321a6ff3c73883afe25fc999ea5/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-20 10:58:50,584 INFO [StoreOpener-96c44321a6ff3c73883afe25fc999ea5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 96c44321a6ff3c73883afe25fc999ea5 2023-03-20 10:58:50,585 DEBUG [StoreOpener-96c44321a6ff3c73883afe25fc999ea5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C 2023-03-20 10:58:50,585 DEBUG [StoreOpener-96c44321a6ff3c73883afe25fc999ea5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C 2023-03-20 10:58:50,585 INFO [StoreOpener-96c44321a6ff3c73883afe25fc999ea5-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-20 10:58:50,585 INFO [StoreOpener-96c44321a6ff3c73883afe25fc999ea5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 96c44321a6ff3c73883afe25fc999ea5 columnFamilyName C 2023-03-20 10:58:50,586 INFO [StoreOpener-96c44321a6ff3c73883afe25fc999ea5-1] regionserver.HStore(310): Store=96c44321a6ff3c73883afe25fc999ea5/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-20 10:58:50,586 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:50,590 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5 2023-03-20 10:58:50,591 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5 2023-03-20 10:58:50,595 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-20 10:58:50,597 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1055): writing seq id for 96c44321a6ff3c73883afe25fc999ea5 2023-03-20 10:58:50,598 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1072): Opened 96c44321a6ff3c73883afe25fc999ea5; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74016270, jitterRate=0.10292837023735046}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-20 10:58:50,598 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(965): Region open journal for 96c44321a6ff3c73883afe25fc999ea5: 2023-03-20 10:58:50,599 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5., pid=61, masterSystemTime=1679309930575 2023-03-20 10:58:50,600 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:50,600 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase20:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:50,600 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=96c44321a6ff3c73883afe25fc999ea5, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:50,600 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679309930600"},{"qualifier":"server","vlen":32,"tag":[],"timestamp":"1679309930600"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679309930600"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679309930600"}]},"ts":"1679309930600"} 2023-03-20 10:58:50,604 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=59 2023-03-20 10:58:50,604 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=59, state=SUCCESS; OpenRegionProcedure 96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 in 178 msec 2023-03-20 10:58:50,605 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=58 2023-03-20 10:58:50,605 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=58, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=96c44321a6ff3c73883afe25fc999ea5, REOPEN/MOVE in 501 msec 2023-03-20 10:58:50,608 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=57 2023-03-20 10:58:50,608 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=57, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 513 msec 2023-03-20 10:58:50,610 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=57, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 931 msec 2023-03-20 10:58:50,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=57 2023-03-20 10:58:50,611 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(139): Connect 0x5ddb3e63 to 127.0.0.1:60226 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 10:58:50,617 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@50131d14, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 10:58:50,618 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(139): Connect 0x630ea023 to 127.0.0.1:60226 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 10:58:50,629 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6722b207, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 10:58:50,630 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(139): Connect 0x5ec98d63 to 127.0.0.1:60226 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 10:58:50,635 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@48502121, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 10:58:50,635 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(139): Connect 0x1ae3ec16 to 127.0.0.1:60226 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 10:58:50,642 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@8d3d8bc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 10:58:50,643 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(139): Connect 0x7bec0400 to 127.0.0.1:60226 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 10:58:50,660 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7e6ad437, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 10:58:50,660 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(139): Connect 0x2f6f7187 to 127.0.0.1:60226 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 10:58:50,671 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@78cd331, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 10:58:50,671 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(139): Connect 0x49c34fbe to 127.0.0.1:60226 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 10:58:50,686 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1d74fc48, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 10:58:50,686 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(139): Connect 0x5467be78 to 127.0.0.1:60226 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 10:58:50,709 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@70ab5f5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 10:58:50,710 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(139): Connect 0x44535519 to 127.0.0.1:60226 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 10:58:50,732 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2daeb795, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 10:58:50,733 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(139): Connect 0x7689435c to 127.0.0.1:60226 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-20 10:58:50,738 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@28ac2b09, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-20 10:58:50,757 DEBUG [hconnection-0x12a33a52-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 10:58:50,757 DEBUG [hconnection-0x44d4c25b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 10:58:50,757 DEBUG [hconnection-0x41a0c4b5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 10:58:50,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:50,759 DEBUG [hconnection-0x2a29198a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 10:58:50,759 DEBUG [hconnection-0x446bc9f4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 10:58:50,759 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:39816, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 10:58:50,760 DEBUG [hconnection-0x9c2bb1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 10:58:50,762 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:39860, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 10:58:50,762 DEBUG [hconnection-0x6926968-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 10:58:50,762 DEBUG [hconnection-0x9fee631-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 10:58:50,764 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:39798, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 10:58:50,764 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:39800, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 10:58:50,767 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:39832, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 10:58:50,765 DEBUG [hconnection-0x76244fcd-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 10:58:50,765 DEBUG [hconnection-0x1cc198a7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-20 10:58:50,767 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:39846, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 10:58:50,771 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:39872, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 10:58:50,773 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:39880, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 10:58:50,776 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:39896, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 10:58:50,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:50,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:50,781 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:50,781 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:50,781 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:50,781 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:50,782 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:50,782 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:50,785 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:50,785 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:50,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:50,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:50,785 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:50,785 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:50,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:50,787 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 148.251.75.209:39904, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-20 10:58:50,787 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:50,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:50,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:50,788 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:50,788 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:50,788 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:50,788 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:50,788 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:50,789 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:50,789 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:50,789 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:50,789 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. started... 2023-03-20 10:58:50,790 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 96c44321a6ff3c73883afe25fc999ea5 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-03-20 10:58:50,790 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 96c44321a6ff3c73883afe25fc999ea5, store=A 2023-03-20 10:58:50,790 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:50,791 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 96c44321a6ff3c73883afe25fc999ea5, store=B 2023-03-20 10:58:50,791 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:50,791 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 96c44321a6ff3c73883afe25fc999ea5, store=C 2023-03-20 10:58:50,791 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:50,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 96c44321a6ff3c73883afe25fc999ea5 2023-03-20 10:58:50,794 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. as already flushing 2023-03-20 10:58:50,830 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230320c863779ab15945ab9ca28748e7d8ab72 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320c863779ab15945ab9ca28748e7d8ab72 2023-03-20 10:58:50,847 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14144, totalHeapSize=32376, min timestamp=1679309930840, max timestamp=1679309930845Num uniques 859; 2023-03-20 10:58:50,847 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=14144, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-20 10:58:50,849 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679309930841, max timestamp=1679309930845Num uniques 859; 2023-03-20 10:58:50,849 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-20 10:58:50,850 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679309930841, max timestamp=1679309930845Num uniques 859; 2023-03-20 10:58:50,850 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-20 10:58:50,852 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=13, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/A/a0e8d5a34c684ebca98a224c039dce91 2023-03-20 10:58:50,885 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1679309930871, max timestamp=1679309930882Num uniques 1168; 2023-03-20 10:58:50,885 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(163): Suffix data size=53488, new segment data size=14558, suffix heap size=149440,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1168, new segment cells count=318 2023-03-20 10:58:50,886 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679309930871, max timestamp=1679309930883Num uniques 1018; 2023-03-20 10:58:50,886 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-20 10:58:50,888 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7682, totalHeapSize=17712, min timestamp=1679309930871, max timestamp=1679309930883Num uniques 1018; 2023-03-20 10:58:50,888 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=46612, new segment data size=7682, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-20 10:58:50,902 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:50,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39798 deadline: 1679309990898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:50,902 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:50,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39832 deadline: 1679309990899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:50,903 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:50,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39816 deadline: 1679309990899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:50,903 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1679309930893, max timestamp=1679309930900Num uniques 1177; 2023-03-20 10:58:50,903 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=53902, new segment data size=10392, suffix heap size=150376,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1177, new segment cells count=227 2023-03-20 10:58:50,903 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1679309930893, max timestamp=1679309930900Num uniques 1027; 2023-03-20 10:58:50,903 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-03-20 10:58:50,904 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1679309930893, max timestamp=1679309930900Num uniques 1027; 2023-03-20 10:58:50,904 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-03-20 10:58:50,930 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:50,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39904 deadline: 1679309990928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:50,932 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:50,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39896 deadline: 1679309990930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,005 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:51,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39816 deadline: 1679309991004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,006 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:51,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39798 deadline: 1679309991004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,007 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:51,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39832 deadline: 1679309991004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,034 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:51,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39904 deadline: 1679309991031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,038 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:51,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39896 deadline: 1679309991036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,209 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:51,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39816 deadline: 1679309991208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,212 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:51,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39832 deadline: 1679309991210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,212 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:51,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39798 deadline: 1679309991211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,239 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:51,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39904 deadline: 1679309991238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,245 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:51,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39896 deadline: 1679309991240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,283 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/B/cef3dd738cbe455aa74b26b8ef844e79 2023-03-20 10:58:51,392 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/C/5753951e6e9743b3a97400013dadc78c 2023-03-20 10:58:51,398 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/A/a0e8d5a34c684ebca98a224c039dce91 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/a0e8d5a34c684ebca98a224c039dce91 2023-03-20 10:58:51,402 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/a0e8d5a34c684ebca98a224c039dce91, entries=150, sequenceid=13, filesize=25.1 K 2023-03-20 10:58:51,402 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/B/cef3dd738cbe455aa74b26b8ef844e79 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/cef3dd738cbe455aa74b26b8ef844e79 2023-03-20 10:58:51,405 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/cef3dd738cbe455aa74b26b8ef844e79, entries=150, sequenceid=13, filesize=11.6 K 2023-03-20 10:58:51,406 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/C/5753951e6e9743b3a97400013dadc78c as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/5753951e6e9743b3a97400013dadc78c 2023-03-20 10:58:51,409 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/5753951e6e9743b3a97400013dadc78c, entries=150, sequenceid=13, filesize=11.6 K 2023-03-20 10:58:51,409 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 96c44321a6ff3c73883afe25fc999ea5 in 619ms, sequenceid=13, compaction requested=false 2023-03-20 10:58:51,409 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-03-20 10:58:51,410 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 96c44321a6ff3c73883afe25fc999ea5: 2023-03-20 10:58:51,410 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:51,410 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:51,410 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:51,410 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,410 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:51,410 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:51,413 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:51,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:51,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:51,413 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:51,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:51,414 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:51,414 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:51,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:51,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:51,416 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:51,416 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75284e65[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:51,416 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:51,416 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:51,417 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:51,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:51,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:51,417 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:51,417 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:51,417 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:51,417 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,417 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:51,418 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:51,418 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:51,418 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:51,418 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:51,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:51,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:51,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:51,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:51,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:51,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:51,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:51,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:51,421 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:51,422 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:51,422 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:51,422 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:51,422 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:51,427 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:51,427 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:51,428 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:51,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:51,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:51,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:51,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:51,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:51,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:51,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:51,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:51,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:51,428 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:51,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:51,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:51,429 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:51,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:51,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,436 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:51,436 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:51,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:51,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:51,436 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:51,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:51,436 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:51,436 DEBUG [Thread-3746] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:51,436 DEBUG [Thread-3746] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:51,436 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:51,436 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:51,436 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,436 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:51,436 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:51,436 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:51,436 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:51,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:51,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:51,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 96c44321a6ff3c73883afe25fc999ea5 2023-03-20 10:58:51,530 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 96c44321a6ff3c73883afe25fc999ea5 3/3 column families, dataSize=161.02 KB heapSize=459 KB 2023-03-20 10:58:51,534 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 96c44321a6ff3c73883afe25fc999ea5, store=A 2023-03-20 10:58:51,534 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-20 10:58:51,534 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 96c44321a6ff3c73883afe25fc999ea5, store=B 2023-03-20 10:58:51,535 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-20 10:58:51,535 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 96c44321a6ff3c73883afe25fc999ea5, store=C 2023-03-20 10:58:51,535 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-20 10:58:51,536 DEBUG [Thread-3746] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:51,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:51,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:51,548 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:51,548 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:51,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39816 deadline: 1679309991543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,549 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:51,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39832 deadline: 1679309991545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,549 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:51,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39798 deadline: 1679309991546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,549 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:51,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39896 deadline: 1679309991547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39904 deadline: 1679309991541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:51,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:51,595 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:51,595 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:51,595 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:51,595 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:51,595 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:51,595 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:51,596 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,596 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:51,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:51,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:51,596 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,597 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:51,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:51,597 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:51,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:51,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:51,597 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:51,628 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:51,628 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:51,628 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:51,628 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:51,628 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:51,640 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:51,640 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:51,641 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:51,641 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. started... 2023-03-20 10:58:51,641 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. as already flushing 2023-03-20 10:58:51,641 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:51,641 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:51,641 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:51,641 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,641 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:51,641 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:51,642 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:51,643 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:51,643 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:51,646 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:51,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:51,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:51,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:51,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:51,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:51,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:51,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@84248ee[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:51,649 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:51,649 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:51,653 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:51,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:51,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:51,653 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:51,653 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:51,653 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:51,653 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,653 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:51,654 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:51,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39816 deadline: 1679309991650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,654 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:51,654 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:51,654 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:51,655 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:51,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39798 deadline: 1679309991650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,655 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:51,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39832 deadline: 1679309991650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,662 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:51,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:51,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:51,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:51,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:51,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:51,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:51,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:51,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:51,668 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:51,668 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:51,668 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:51,668 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:51,668 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:51,672 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230320b6979c8e9357488aa74ad04b575d22c0 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320b6979c8e9357488aa74ad04b575d22c0 2023-03-20 10:58:51,674 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:51,674 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:51,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:51,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:51,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:51,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:51,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:51,675 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:51,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:51,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:51,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:51,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:51,676 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:51,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:51,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:51,682 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:51,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:51,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,702 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:51,702 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:51,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:51,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:51,702 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:51,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:51,702 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:51,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:51,703 DEBUG [Thread-3746] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:51,703 DEBUG [Thread-3746] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:51,706 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,706 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:51,706 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:51,706 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,706 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:51,706 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:51,706 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:51,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:51,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:51,803 DEBUG [Thread-3746] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:51,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:51,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:51,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:51,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:51,844 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:51,844 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:51,844 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:51,844 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:51,844 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:51,844 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:51,845 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:51,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:51,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:51,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:51,845 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:51,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:51,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:51,846 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:51,846 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,846 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:51,846 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:51,846 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:51,846 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:51,846 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:51,847 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,847 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:51,859 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:51,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39832 deadline: 1679309991856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,860 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:51,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39816 deadline: 1679309991856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,860 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:51,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39798 deadline: 1679309991857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,861 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:51,862 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:51,862 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:51,862 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. started... 2023-03-20 10:58:51,862 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. as already flushing 2023-03-20 10:58:51,862 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:51,862 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:51,862 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:51,862 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,862 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:51,862 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:51,863 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:51,863 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:51,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:51,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:51,864 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:51,864 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:51,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:51,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:51,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:51,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:51,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4dcf0f49[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:51,866 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:51,866 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:51,867 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:51,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:51,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:51,868 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,868 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:51,868 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:51,868 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:51,868 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:51,872 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,872 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:51,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:51,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:51,872 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:51,872 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:51,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:51,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:51,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:51,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:51,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:51,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:51,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:51,876 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:51,876 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:51,876 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:51,876 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:51,876 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:51,886 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:51,886 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:51,886 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:51,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:51,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:51,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:51,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:51,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:51,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:51,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:51,886 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:51,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:51,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:51,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:51,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:51,888 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:51,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:51,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,890 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,890 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:51,890 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:51,890 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:51,890 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:51,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:51,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:51,890 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:51,890 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:51,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:51,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:51,890 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,890 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:51,890 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:51,890 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:51,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:51,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:51,890 DEBUG [Thread-3746] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:51,890 DEBUG [Thread-3746] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:51,991 DEBUG [Thread-3746] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:51,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:51,992 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:51,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:51,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:51,993 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:51,993 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:51,993 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:51,993 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:51,994 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:51,994 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:51,994 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:51,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:51,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:51,994 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:51,995 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,995 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:51,995 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:51,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:51,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:51,995 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:51,995 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:51,995 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:51,995 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:51,995 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:51,996 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:51,996 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:51,996 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:51,996 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:51,996 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. started... 2023-03-20 10:58:51,996 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. as already flushing 2023-03-20 10:58:51,996 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:51,996 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:51,996 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:51,996 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,996 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:51,996 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:51,997 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,997 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:51,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:51,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:51,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:51,997 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:51,997 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:51,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:51,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:51,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:51,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:51,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c8b8b03[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:51,998 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:51,998 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:51,998 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:51,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:51,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:51,999 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:51,999 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:51,999 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:51,999 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,999 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:51,999 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,999 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:51,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:51,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:51,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:51,999 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:51,999 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:52,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:52,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:52,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:52,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:52,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:52,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:52,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:52,002 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:52,002 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:52,002 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:52,002 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:52,002 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:52,003 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:52,003 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:52,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:52,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:52,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:52,003 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:52,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:52,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:52,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:52,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:52,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:52,003 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:52,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:52,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:52,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:52,004 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:52,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:52,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,008 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,008 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:52,008 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:52,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:52,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:52,008 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:52,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:52,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:52,008 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:52,008 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:52,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:52,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:52,008 DEBUG [Thread-3746] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:52,008 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:52,008 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,009 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:52,009 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:52,009 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:52,009 DEBUG [Thread-3746] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:52,055 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:52,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39904 deadline: 1679309992053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,056 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:52,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39896 deadline: 1679309992054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,110 DEBUG [Thread-3746] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:52,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:52,116 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=87, memsize=53.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/A/7ae3e523a54e4b27bcb77bdb6a21c577 2023-03-20 10:58:52,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:52,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:52,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:52,150 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:52,150 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:52,150 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:52,150 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:52,158 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:52,158 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:52,159 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,160 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,160 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:52,161 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:52,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:52,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:52,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:52,162 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:52,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:52,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:52,162 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:52,166 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:52,166 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:52,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39816 deadline: 1679309992162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39832 deadline: 1679309992162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,167 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:52,167 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:52,167 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:52,168 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:52,168 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:52,170 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:52,170 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:52,173 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:52,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39798 deadline: 1679309992168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,174 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:52,174 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. started... 2023-03-20 10:58:52,174 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. as already flushing 2023-03-20 10:58:52,174 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:52,175 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:52,175 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:52,175 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,175 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:52,175 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:52,186 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:52,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:52,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:52,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:52,192 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:52,192 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:52,192 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:52,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:52,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:52,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:52,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5be7bb91[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:52,194 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:52,194 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:52,224 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:52,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:52,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:52,224 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,224 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:52,224 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:52,224 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:52,224 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:52,225 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:52,225 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,225 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:52,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:52,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:52,225 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:52,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:52,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:52,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:52,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:52,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:52,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:52,227 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:52,227 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:52,227 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:52,227 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:52,227 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:52,227 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:52,228 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:52,228 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:52,228 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:52,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:52,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:52,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:52,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:52,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:52,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:52,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:52,229 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:52,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:52,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:52,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:52,230 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:52,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:52,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:52,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,252 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,252 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:52,252 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:52,253 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,253 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:52,253 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:52,253 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:52,253 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:52,253 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:52,253 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:52,253 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:52,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:52,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:52,253 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:52,253 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:52,253 DEBUG [Thread-3746] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:52,253 DEBUG [Thread-3746] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:52,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:52,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:52,353 DEBUG [Thread-3746] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:52,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:52,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:52,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:52,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:52,366 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:52,366 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:52,366 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:52,366 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:52,367 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:52,367 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:52,368 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:52,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:52,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:52,368 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:52,368 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:52,369 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,369 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:52,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:52,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:52,369 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:52,370 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:52,370 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:52,370 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:52,371 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:52,371 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:52,374 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:52,374 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:52,374 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:52,374 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. started... 2023-03-20 10:58:52,374 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. as already flushing 2023-03-20 10:58:52,374 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:52,374 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:52,374 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:52,374 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,374 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:52,374 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:52,377 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,377 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:52,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:52,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:52,378 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:52,378 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:52,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:52,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:52,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:52,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:52,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:52,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41f88d3b[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:52,379 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:52,379 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:52,380 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:52,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:52,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:52,380 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:52,380 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:52,380 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:52,380 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,380 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:52,381 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:52,381 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:52,381 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,381 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:52,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:52,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:52,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:52,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:52,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:52,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:52,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:52,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:52,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:52,384 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:52,384 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:52,384 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:52,384 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:52,384 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:52,385 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:52,385 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:52,385 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:52,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:52,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:52,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:52,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:52,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:52,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:52,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:52,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:52,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:52,386 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:52,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:52,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:52,387 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:52,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:52,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,388 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:52,388 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:52,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:52,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:52,389 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:52,389 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,389 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:52,389 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:52,389 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:52,389 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:52,389 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,389 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:52,389 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:52,389 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:52,389 DEBUG [Thread-3746] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:52,389 DEBUG [Thread-3746] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:52,389 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:52,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:52,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:52,489 DEBUG [Thread-3746] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:52,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:52,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:52,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:52,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:52,491 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:52,492 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:52,492 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:52,492 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:52,492 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:52,492 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:52,493 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:52,493 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:52,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:52,493 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,493 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:52,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:52,493 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:52,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:52,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:52,494 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:52,494 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:52,494 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:52,494 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:52,494 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:52,494 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:52,494 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:52,494 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:52,495 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:52,495 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. started... 2023-03-20 10:58:52,495 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. as already flushing 2023-03-20 10:58:52,495 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:52,495 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:52,495 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:52,495 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,495 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:52,495 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:52,496 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:52,496 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:52,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:52,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:52,496 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:52,496 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:52,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:52,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:52,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:52,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:52,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74f8770a[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:52,497 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:52,497 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:52,498 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:52,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:52,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:52,498 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:52,498 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:52,498 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:52,498 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,498 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:52,499 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:52,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:52,499 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:52,499 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:52,499 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:52,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:52,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:52,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:52,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:52,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:52,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:52,501 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:52,501 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:52,501 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:52,501 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:52,501 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:52,501 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:52,503 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:52,503 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:52,503 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:52,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:52,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:52,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:52,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:52,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:52,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:52,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:52,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:52,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:52,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:52,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:52,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:52,505 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:52,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:52,505 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,509 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:52,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:52,509 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:52,509 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:52,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:52,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:52,510 DEBUG [Thread-3746] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:52,510 DEBUG [Thread-3746] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:52,509 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:52,509 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:52,509 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:52,510 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:52,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:52,510 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:52,510 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:52,510 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:52,579 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=87 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/B/537ae667411d48bda05589632378b39b 2023-03-20 10:58:52,611 DEBUG [Thread-3746] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:52,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:52,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:52,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:52,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:52,649 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:52,649 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:52,649 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:52,649 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:52,649 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:52,649 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:52,650 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:52,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:52,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:52,650 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:52,651 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:52,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:52,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:52,651 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:52,651 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=87 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/C/7c50fcf180334f9aae90b0d6a456cfe5 2023-03-20 10:58:52,651 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,652 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:52,656 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/A/7ae3e523a54e4b27bcb77bdb6a21c577 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/7ae3e523a54e4b27bcb77bdb6a21c577 2023-03-20 10:58:52,661 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:52,661 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:52,662 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:52,662 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:52,662 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:52,664 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/7ae3e523a54e4b27bcb77bdb6a21c577, entries=150, sequenceid=87, filesize=25.1 K 2023-03-20 10:58:52,665 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/B/537ae667411d48bda05589632378b39b as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/537ae667411d48bda05589632378b39b 2023-03-20 10:58:52,666 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:52,666 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:52,670 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:52,670 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. started... 2023-03-20 10:58:52,670 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. as already flushing 2023-03-20 10:58:52,670 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:52,670 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:52,670 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:52,670 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,670 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:52,670 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:52,672 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:52,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39832 deadline: 1679309992668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,674 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/537ae667411d48bda05589632378b39b, entries=150, sequenceid=87, filesize=11.6 K 2023-03-20 10:58:52,675 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/C/7c50fcf180334f9aae90b0d6a456cfe5 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/7c50fcf180334f9aae90b0d6a456cfe5 2023-03-20 10:58:52,677 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:52,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39816 deadline: 1679309992668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,684 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/7c50fcf180334f9aae90b0d6a456cfe5, entries=150, sequenceid=87, filesize=11.6 K 2023-03-20 10:58:52,687 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:52,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:52,687 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:52,688 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~161.02 KB/164880, heapSize ~458.95 KB/469968, currentSize=20.13 KB/20610 for 96c44321a6ff3c73883afe25fc999ea5 in 1158ms, sequenceid=87, compaction requested=false 2023-03-20 10:58:52,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:52,689 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:52,689 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:52,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:52,690 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 96c44321a6ff3c73883afe25fc999ea5: 2023-03-20 10:58:52,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:52,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:52,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:52,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@441b2905[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:52,699 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:52,699 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:52,700 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:52,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:52,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:52,700 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,700 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:52,700 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:52,700 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:52,700 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:52,701 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,701 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:52,701 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:52,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:52,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:52,701 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:52,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:52,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:52,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:52,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:52,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:52,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:52,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:52,708 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:52,708 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:52,708 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:52,708 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:52,708 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:52,711 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:52,711 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:52,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:52,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:52,711 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:52,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:52,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:52,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:52,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:52,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:52,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:52,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:52,712 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:52,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:52,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:52,714 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:52,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:52,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,718 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 96c44321a6ff3c73883afe25fc999ea5 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-20 10:58:52,718 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 96c44321a6ff3c73883afe25fc999ea5, store=A 2023-03-20 10:58:52,718 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:52,718 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 96c44321a6ff3c73883afe25fc999ea5, store=B 2023-03-20 10:58:52,718 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:52,718 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 96c44321a6ff3c73883afe25fc999ea5, store=C 2023-03-20 10:58:52,719 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:52,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 96c44321a6ff3c73883afe25fc999ea5 2023-03-20 10:58:52,719 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:52,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:52,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:52,720 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:52,720 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:52,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:52,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:52,720 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,720 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:52,720 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:52,720 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,720 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:52,721 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:52,721 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:52,721 DEBUG [Thread-3746] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:52,721 DEBUG [Thread-3746] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:52,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:52,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:52,722 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:52,821 DEBUG [Thread-3746] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:52,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:52,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:52,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:52,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:52,832 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:52,832 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:52,832 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:52,832 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:52,832 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:52,832 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:52,833 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,833 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:52,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:52,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:52,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:52,834 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,834 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:52,834 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:52,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:52,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:52,835 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:52,838 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:52,838 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:52,838 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:52,838 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:52,838 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:52,842 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:52,842 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:52,842 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:52,842 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. started... 2023-03-20 10:58:52,842 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. as already flushing 2023-03-20 10:58:52,842 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:52,842 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:52,842 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:52,842 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,842 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:52,842 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:52,844 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:52,844 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:52,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:52,845 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:52,845 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:52,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:52,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:52,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:52,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:52,847 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:52,847 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22463924[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:52,847 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:52,847 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:52,848 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:52,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:52,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:52,848 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:52,848 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:52,848 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:52,848 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,848 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:52,849 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:52,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:52,850 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679309932761, max timestamp=1679309932848Num uniques 859; 2023-03-20 10:58:52,850 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-20 10:58:52,851 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:52,851 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:52,851 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:52,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:52,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:52,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:52,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:52,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:52,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:52,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:52,853 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:52,853 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:52,853 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:52,853 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:52,853 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:52,855 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:52,855 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:52,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:52,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:52,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:52,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:52,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:52,856 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:52,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:52,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:52,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:52,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:52,857 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:52,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:52,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:52,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:52,859 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:52,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,861 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:52,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:52,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:52,861 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,861 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:52,861 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:52,861 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,861 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:52,861 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:52,861 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:52,862 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:52,861 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:52,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:52,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:52,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:52,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:52,862 DEBUG [Thread-3746] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:52,862 DEBUG [Thread-3746] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:52,895 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:52,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39798 deadline: 1679309992892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,962 DEBUG [Thread-3746] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:52,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:52,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:52,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:52,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:52,978 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:52,978 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:52,978 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:52,978 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:52,979 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:52,979 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:52,979 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:52,979 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:52,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:52,980 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,980 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:52,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:52,980 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:52,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:52,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:52,980 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:52,982 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:52,982 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:52,982 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:52,982 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:52,982 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:52,986 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:52,986 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:52,990 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:52,990 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. started... 2023-03-20 10:58:52,990 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. as already flushing 2023-03-20 10:58:52,990 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:52,990 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:52,990 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:52,990 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:52,990 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:52,990 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:53,002 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:53,003 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:53,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:53,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:53,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:53,003 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:53,003 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:53,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:53,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:53,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:53,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:53,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:53,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:53,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2752e902[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:53,004 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:53,004 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:53,005 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:53,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:53,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:53,005 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:53,005 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:53,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39798 deadline: 1679309993002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:53,005 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:53,007 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:53,005 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:53,008 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:53,011 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:53,011 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:53,011 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:53,012 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:53,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:53,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:53,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:53,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:53,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:53,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:53,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:53,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:53,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:53,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:53,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:53,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:53,014 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:53,014 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:53,014 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:53,014 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:53,014 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:53,015 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:53,015 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:53,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:53,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:53,016 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:53,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:53,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:53,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:53,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:53,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:53,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:53,017 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:53,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:53,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:53,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:53,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:53,018 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:53,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:53,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:53,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:53,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:53,021 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:53,021 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:53,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:53,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:53,021 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:53,021 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:53,021 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:53,021 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:53,021 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:53,021 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:53,021 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:53,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:53,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:53,021 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:53,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:53,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:53,022 DEBUG [Thread-3746] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:53,022 DEBUG [Thread-3746] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:53,022 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:53,060 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:53,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39904 deadline: 1679309993059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:53,066 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:53,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39896 deadline: 1679309993063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:53,124 DEBUG [Thread-3746] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:53,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:53,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:53,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:53,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:53,138 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:53,138 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:53,138 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:53,138 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:53,138 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:53,139 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:53,139 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:53,139 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:53,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:53,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:53,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:53,140 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:53,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:53,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:53,140 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:53,141 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:53,141 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:53,141 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:53,141 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:53,141 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:53,141 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:53,141 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:53,142 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:53,142 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:53,142 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:53,142 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. started... 2023-03-20 10:58:53,142 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. as already flushing 2023-03-20 10:58:53,142 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:53,142 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:53,142 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:53,142 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:53,142 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:53,142 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:53,144 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:53,144 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:53,144 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:53,146 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:53,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:53,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:53,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:53,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:53,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:53,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:53,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:53,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:53,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:53,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10258ac5[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:53,147 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:53,148 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:53,148 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:53,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:53,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:53,148 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:53,148 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:53,148 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:53,149 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:53,149 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:53,154 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:53,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:53,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:53,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:53,154 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:53,154 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:53,154 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:53,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:53,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:53,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:53,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:53,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:53,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:53,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:53,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:53,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:53,160 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:53,160 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:53,160 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:53,160 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:53,160 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:53,163 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230320c238bc5caef1436d8c125abc2499a078 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320c238bc5caef1436d8c125abc2499a078 2023-03-20 10:58:53,175 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:53,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:53,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:53,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:53,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:53,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:53,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:53,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:53,177 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:53,177 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:53,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:53,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:53,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:53,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:53,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:53,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:53,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:53,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:53,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:53,182 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:53,200 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:53,204 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=97, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/A/e35e127666fd416e9a8ab32872b47db4 2023-03-20 10:58:53,207 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:53,207 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:53,207 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:53,207 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:53,207 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:53,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:53,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:53,208 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:53,208 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:53,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:53,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:53,207 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:53,208 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:53,208 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:53,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:53,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:53,212 DEBUG [Thread-3746] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:53,212 DEBUG [Thread-3746] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:53,213 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:53,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39798 deadline: 1679309993210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:53,314 DEBUG [Thread-3746] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:53,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:53,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:53,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:53,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:53,339 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:53,339 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:53,339 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:53,339 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:53,352 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:53,352 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:53,358 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:53,359 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:53,359 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:53,362 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:53,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:53,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:53,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:53,365 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:53,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:53,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:53,365 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:53,366 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:53,366 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:53,366 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:53,366 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:53,366 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:53,366 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:53,366 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:53,366 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:53,366 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. started... 2023-03-20 10:58:53,366 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. as already flushing 2023-03-20 10:58:53,366 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:53,367 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:53,367 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:53,367 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:53,367 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:53,367 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:53,378 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:53,380 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:53,380 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:53,381 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:53,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:53,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:53,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:53,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:53,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:53,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:53,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:53,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:53,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:53,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d6bdac5[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:53,386 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:53,386 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:53,390 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:53,391 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:53,393 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:53,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:53,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:53,393 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:53,393 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:53,393 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:53,407 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:53,407 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:53,407 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:53,410 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:53,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:53,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:53,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:53,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:53,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:53,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:53,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:53,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:53,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:53,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:53,461 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-20 10:58:53,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:53,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:53,469 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:53,469 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:53,469 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:53,469 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:53,469 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:53,494 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:53,494 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:53,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:53,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:53,498 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:53,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:53,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:53,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:53,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:53,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:53,504 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:53,505 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:53,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:53,515 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:53,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:53,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:53,515 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:53,515 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:53,515 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:53,515 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:53,515 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:53,515 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:53,515 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:53,515 DEBUG [Thread-3746] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:53,515 DEBUG [Thread-3746] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:53,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:53,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:53,529 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:53,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:53,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:53,533 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:53,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39798 deadline: 1679309993531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:53,534 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:53,615 DEBUG [Thread-3746] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:53,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:53,623 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:53,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:53,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:53,632 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:53,632 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:53,632 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:53,632 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:53,633 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:53,634 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:53,643 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:53,643 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:53,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:53,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:53,647 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=97 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/B/e943679fc0494b84b27bee7dfcff505c 2023-03-20 10:58:53,647 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:53,647 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:53,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:53,648 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:53,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:53,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:53,648 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:53,664 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:53,664 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:53,664 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:53,664 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:53,664 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:53,676 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:53,676 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:53,677 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:53,677 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. started... 2023-03-20 10:58:53,677 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. as already flushing 2023-03-20 10:58:53,677 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:53,678 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:53,678 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:53,678 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:53,678 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:53,678 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:53,692 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:53,693 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:53,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39816 deadline: 1679309993682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:53,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39832 deadline: 1679309993683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:53,711 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:53,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:53,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:53,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:53,714 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:53,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:53,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:53,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:53,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:53,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:53,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:53,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e774d55[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:53,716 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:53,716 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:53,718 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:53,718 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:53,730 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:53,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:53,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:53,730 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:53,730 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:53,730 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:53,738 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:53,738 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:53,754 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:53,754 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:53,754 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:53,760 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:53,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:53,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:53,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:53,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:53,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:53,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:53,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:53,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:53,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:53,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:53,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:53,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:53,790 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:53,790 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:53,790 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:53,790 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:53,790 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:53,798 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:53,798 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:53,798 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:53,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:53,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:53,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:53,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:53,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:53,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:53,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:53,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:53,810 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:53,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:53,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:53,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:53,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:53,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:53,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:53,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:53,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:53,836 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:53,853 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:53,853 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:53,853 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:53,853 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:53,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:53,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:53,853 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:53,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:53,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:53,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:53,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:53,853 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:53,854 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:53,854 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:53,854 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:53,854 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:53,854 DEBUG [Thread-3746] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:53,854 DEBUG [Thread-3746] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:53,955 DEBUG [Thread-3746] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:53,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:53,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:53,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:53,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:53,970 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:53,971 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:53,971 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:53,971 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:53,974 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:53,974 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:53,975 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:53,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:53,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:53,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:53,976 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:53,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:53,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:53,976 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:53,978 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:53,978 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:53,978 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:53,978 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:53,978 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:53,978 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:53,981 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:53,981 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:53,990 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:53,990 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:53,999 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:54,000 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. started... 2023-03-20 10:58:54,000 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. as already flushing 2023-03-20 10:58:54,000 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:54,000 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:54,000 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:54,000 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:54,000 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:54,000 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:54,006 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:54,006 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:54,007 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:54,007 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:54,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:54,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:54,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:54,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:54,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:54,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:54,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:54,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:54,018 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:54,018 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2eddbd60[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:54,018 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:54,018 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:54,027 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:54,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:54,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:54,027 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:54,027 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:54,027 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:54,038 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:54,038 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:54,038 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:54,040 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:54,043 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:54,043 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:54,043 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:54,044 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:54,044 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:54,044 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:54,044 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:54,045 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:54,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39798 deadline: 1679309994041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:54,059 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:54,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:54,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:54,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:54,062 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:54,062 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:54,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:54,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:54,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:54,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:54,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:54,066 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:54,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:54,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:54,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:54,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:54,094 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:54,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:54,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:54,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:54,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:54,126 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:54,126 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:54,127 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:54,127 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:54,127 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:54,127 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:54,127 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:54,127 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:54,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:54,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:54,128 DEBUG [Thread-3746] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:54,128 DEBUG [Thread-3746] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:54,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:54,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:54,127 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:54,130 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:54,130 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:54,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:54,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:54,145 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=97 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/C/d819f9d6802646d89c280a75b5e4e7fa 2023-03-20 10:58:54,167 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/A/e35e127666fd416e9a8ab32872b47db4 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/e35e127666fd416e9a8ab32872b47db4 2023-03-20 10:58:54,170 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/e35e127666fd416e9a8ab32872b47db4, entries=150, sequenceid=97, filesize=25.1 K 2023-03-20 10:58:54,171 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/B/e943679fc0494b84b27bee7dfcff505c as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/e943679fc0494b84b27bee7dfcff505c 2023-03-20 10:58:54,182 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/e943679fc0494b84b27bee7dfcff505c, entries=150, sequenceid=97, filesize=11.6 K 2023-03-20 10:58:54,184 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/C/d819f9d6802646d89c280a75b5e4e7fa as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/d819f9d6802646d89c280a75b5e4e7fa 2023-03-20 10:58:54,192 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/d819f9d6802646d89c280a75b5e4e7fa, entries=150, sequenceid=97, filesize=11.6 K 2023-03-20 10:58:54,196 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=129.71 KB/132820 for 96c44321a6ff3c73883afe25fc999ea5 in 1478ms, sequenceid=97, compaction requested=true 2023-03-20 10:58:54,196 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 96c44321a6ff3c73883afe25fc999ea5: 2023-03-20 10:58:54,196 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:58:54,196 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:58:54,196 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 10:58:54,196 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:58:54,196 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 10:58:54,203 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 77238 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 10:58:54,203 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1912): 96c44321a6ff3c73883afe25fc999ea5/A is initiating minor compaction (all files) 2023-03-20 10:58:54,203 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 96c44321a6ff3c73883afe25fc999ea5/A in TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:54,203 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/a0e8d5a34c684ebca98a224c039dce91, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/7ae3e523a54e4b27bcb77bdb6a21c577, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/e35e127666fd416e9a8ab32872b47db4] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp, totalSize=75.4 K 2023-03-20 10:58:54,203 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting a0e8d5a34c684ebca98a224c039dce91, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1679309930787 2023-03-20 10:58:54,204 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 7ae3e523a54e4b27bcb77bdb6a21c577, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=87, earliestPutTs=1679309930922 2023-03-20 10:58:54,203 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 10:58:54,204 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): 96c44321a6ff3c73883afe25fc999ea5/B is initiating minor compaction (all files) 2023-03-20 10:58:54,204 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 96c44321a6ff3c73883afe25fc999ea5/B in TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:54,204 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/cef3dd738cbe455aa74b26b8ef844e79, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/537ae667411d48bda05589632378b39b, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/e943679fc0494b84b27bee7dfcff505c] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp, totalSize=34.8 K 2023-03-20 10:58:54,204 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting e35e127666fd416e9a8ab32872b47db4, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1679309931538 2023-03-20 10:58:54,204 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting cef3dd738cbe455aa74b26b8ef844e79, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1679309930787 2023-03-20 10:58:54,206 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 537ae667411d48bda05589632378b39b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=87, earliestPutTs=1679309930922 2023-03-20 10:58:54,206 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting e943679fc0494b84b27bee7dfcff505c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1679309931538 2023-03-20 10:58:54,224 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] throttle.PressureAwareThroughputController(145): 96c44321a6ff3c73883afe25fc999ea5#A#compaction#511 average throughput is 1.97 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:58:54,229 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): 96c44321a6ff3c73883afe25fc999ea5#B#compaction#512 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:58:54,230 DEBUG [Thread-3746] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:54,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:54,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:54,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:54,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:54,246 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:54,246 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:54,246 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:54,246 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:54,248 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:54,248 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:54,259 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:54,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:54,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:54,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:54,259 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:54,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:54,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:54,260 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:54,262 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:54,262 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:54,262 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:54,288 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:54,288 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:54,288 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:54,288 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:54,288 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:54,303 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:54,303 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:54,306 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:54,306 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. started... 2023-03-20 10:58:54,306 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 96c44321a6ff3c73883afe25fc999ea5 3/3 column families, dataSize=129.71 KB heapSize=381.64 KB 2023-03-20 10:58:54,306 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 96c44321a6ff3c73883afe25fc999ea5, store=A 2023-03-20 10:58:54,306 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:58:54,306 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 96c44321a6ff3c73883afe25fc999ea5, store=B 2023-03-20 10:58:54,307 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:58:54,307 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 96c44321a6ff3c73883afe25fc999ea5, store=C 2023-03-20 10:58:54,307 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:58:54,394 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032025cbb804680a4b9e9a04a7ab57855d6e to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032025cbb804680a4b9e9a04a7ab57855d6e 2023-03-20 10:58:54,437 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=124, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/A/56a19abb20c841a18893eed3c7781d1a 2023-03-20 10:58:54,750 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/B/ac5c7965469f42a7b932280c5c57d2fa as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/ac5c7965469f42a7b932280c5c57d2fa 2023-03-20 10:58:54,791 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 96c44321a6ff3c73883afe25fc999ea5/B of 96c44321a6ff3c73883afe25fc999ea5 into ac5c7965469f42a7b932280c5c57d2fa(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:58:54,791 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 96c44321a6ff3c73883afe25fc999ea5: 2023-03-20 10:58:54,791 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5., storeName=96c44321a6ff3c73883afe25fc999ea5/B, priority=13, startTime=1679309934196; duration=0sec 2023-03-20 10:58:54,791 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:58:54,791 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 10:58:54,806 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 10:58:54,806 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): 96c44321a6ff3c73883afe25fc999ea5/C is initiating minor compaction (all files) 2023-03-20 10:58:54,806 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 96c44321a6ff3c73883afe25fc999ea5/C in TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:54,833 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/5753951e6e9743b3a97400013dadc78c, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/7c50fcf180334f9aae90b0d6a456cfe5, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/d819f9d6802646d89c280a75b5e4e7fa] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp, totalSize=34.8 K 2023-03-20 10:58:54,838 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 5753951e6e9743b3a97400013dadc78c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1679309930787 2023-03-20 10:58:54,839 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 7c50fcf180334f9aae90b0d6a456cfe5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=87, earliestPutTs=1679309930922 2023-03-20 10:58:54,839 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting d819f9d6802646d89c280a75b5e4e7fa, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1679309931538 2023-03-20 10:58:54,855 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): 96c44321a6ff3c73883afe25fc999ea5#C#compaction#515 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:58:54,859 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/A/d17893d5991a4ae2a44caec61d1385ca as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/d17893d5991a4ae2a44caec61d1385ca 2023-03-20 10:58:54,867 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 96c44321a6ff3c73883afe25fc999ea5/A of 96c44321a6ff3c73883afe25fc999ea5 into d17893d5991a4ae2a44caec61d1385ca(size=25.2 K), total size for store is 25.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:58:54,867 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 96c44321a6ff3c73883afe25fc999ea5: 2023-03-20 10:58:54,867 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5., storeName=96c44321a6ff3c73883afe25fc999ea5/A, priority=13, startTime=1679309934196; duration=0sec 2023-03-20 10:58:54,867 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:58:54,879 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/C/349b1558bd3e44d3b8fa0359525c479d as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/349b1558bd3e44d3b8fa0359525c479d 2023-03-20 10:58:54,885 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 96c44321a6ff3c73883afe25fc999ea5/C of 96c44321a6ff3c73883afe25fc999ea5 into 349b1558bd3e44d3b8fa0359525c479d(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:58:54,885 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 96c44321a6ff3c73883afe25fc999ea5: 2023-03-20 10:58:54,885 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5., storeName=96c44321a6ff3c73883afe25fc999ea5/C, priority=13, startTime=1679309934196; duration=0sec 2023-03-20 10:58:54,885 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:58:54,923 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=124 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/B/8358210835f8404f9d814c6f0867bebe 2023-03-20 10:58:54,956 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=124 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/C/89696bd472024b7f90144851f8723817 2023-03-20 10:58:54,961 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/A/56a19abb20c841a18893eed3c7781d1a as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/56a19abb20c841a18893eed3c7781d1a 2023-03-20 10:58:54,967 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/56a19abb20c841a18893eed3c7781d1a, entries=150, sequenceid=124, filesize=25.1 K 2023-03-20 10:58:54,968 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/B/8358210835f8404f9d814c6f0867bebe as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/8358210835f8404f9d814c6f0867bebe 2023-03-20 10:58:54,972 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/8358210835f8404f9d814c6f0867bebe, entries=150, sequenceid=124, filesize=11.6 K 2023-03-20 10:58:54,972 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/C/89696bd472024b7f90144851f8723817 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/89696bd472024b7f90144851f8723817 2023-03-20 10:58:54,977 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/89696bd472024b7f90144851f8723817, entries=150, sequenceid=124, filesize=11.6 K 2023-03-20 10:58:54,978 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~129.71 KB/132820, heapSize ~381.59 KB/390752, currentSize=0 B/0 for 96c44321a6ff3c73883afe25fc999ea5 in 672ms, sequenceid=124, compaction requested=false 2023-03-20 10:58:54,978 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 96c44321a6ff3c73883afe25fc999ea5: 2023-03-20 10:58:54,978 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:54,978 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:54,978 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:54,978 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:54,978 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:54,978 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:54,979 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:54,979 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:54,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:54,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:54,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:54,980 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:54,980 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:54,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:54,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:54,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:54,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:54,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:54,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:54,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@129269dc[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:54,981 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:54,981 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:54,982 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:54,982 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:54,982 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:54,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:54,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:54,982 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:54,982 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:54,982 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:54,983 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:54,983 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:54,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:54,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:54,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:54,983 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:54,983 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:54,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:54,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:54,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:54,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:54,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:54,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:54,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:54,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:54,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:54,985 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:54,986 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:54,986 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:54,986 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:54,986 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:54,987 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:54,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:54,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:54,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:54,987 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:54,987 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:54,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:54,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:54,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:54,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:54,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:54,988 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:54,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:54,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:54,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:54,990 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:54,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:54,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:54,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:54,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:54,994 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:54,994 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:54,994 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:54,994 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:54,994 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:54,994 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:54,994 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:54,994 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:54,994 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:54,994 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:54,995 DEBUG [Thread-3746] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:54,995 DEBUG [Thread-3746] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:54,994 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:54,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:54,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:54,994 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:54,995 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:54,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:54,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:55,097 DEBUG [Thread-3746] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:55,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:55,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:55,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 96c44321a6ff3c73883afe25fc999ea5 2023-03-20 10:58:55,109 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 96c44321a6ff3c73883afe25fc999ea5 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-20 10:58:55,109 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 96c44321a6ff3c73883afe25fc999ea5, store=A 2023-03-20 10:58:55,109 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:55,109 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 96c44321a6ff3c73883afe25fc999ea5, store=B 2023-03-20 10:58:55,109 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:55,109 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 96c44321a6ff3c73883afe25fc999ea5, store=C 2023-03-20 10:58:55,109 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:55,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:55,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:55,114 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:55,114 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:55,114 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:55,114 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:55,115 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:55,115 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:55,129 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:55,130 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:55,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:55,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:55,130 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:55,130 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:55,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:55,131 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:55,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:55,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:55,132 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:55,132 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:55,132 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:55,132 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:55,132 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:55,132 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:55,133 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:55,133 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:55,136 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:55,136 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. started... 2023-03-20 10:58:55,136 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. as already flushing 2023-03-20 10:58:55,136 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:55,138 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303206964d499908744ee895fcf8fd273509f to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303206964d499908744ee895fcf8fd273509f 2023-03-20 10:58:55,146 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:55,146 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:55,146 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:55,146 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:55,146 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:55,149 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:55,149 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:55,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:55,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:55,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:55,149 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:55,149 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:55,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:55,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:55,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:55,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:55,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:55,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:55,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@417c24b9[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:55,151 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:55,151 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:55,152 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:55,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:55,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:55,152 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:55,152 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:55,152 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:55,152 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:55,152 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:55,153 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:55,153 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:55,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:55,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:55,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:55,153 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:55,153 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:55,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:55,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:55,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:55,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:55,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:55,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:55,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:55,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:55,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:55,156 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:55,156 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:55,156 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:55,156 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:55,156 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:55,157 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:55,157 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:55,157 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:55,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:55,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:55,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:55,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:55,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:55,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:55,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:55,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:55,159 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:55,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:55,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:55,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:55,161 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:55,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:55,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:55,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:55,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:55,165 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:55,165 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:55,165 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:55,165 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:55,165 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:55,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:55,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:55,165 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:55,166 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:55,166 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:55,166 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:55,166 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:55,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:55,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:55,166 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:55,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:55,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:55,168 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679309935120, max timestamp=1679309935165Num uniques 859; 2023-03-20 10:58:55,168 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-20 10:58:55,169 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=137, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/A/576b677cd9ad4a5b8cfcc97aea28753a 2023-03-20 10:58:55,169 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679309935157, max timestamp=1679309935165Num uniques 859; 2023-03-20 10:58:55,169 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-20 10:58:55,170 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679309935157, max timestamp=1679309935165Num uniques 859; 2023-03-20 10:58:55,170 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-20 10:58:55,174 DEBUG [Thread-3746] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:55,174 DEBUG [Thread-3746] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:55,236 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1679309935213, max timestamp=1679309935233Num uniques 1118; 2023-03-20 10:58:55,236 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=12268, suffix heap size=144240,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=268 2023-03-20 10:58:55,236 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1679309935213, max timestamp=1679309935233Num uniques 1018; 2023-03-20 10:58:55,236 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-03-20 10:58:55,238 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1679309935213, max timestamp=1679309935233Num uniques 1018; 2023-03-20 10:58:55,238 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=10760, suffix heap size=133840,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=235 2023-03-20 10:58:55,278 DEBUG [Thread-3746] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:55,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:55,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:55,304 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:55,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39904 deadline: 1679309995299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:55,304 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:55,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39896 deadline: 1679309995299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:55,305 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:55,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39798 deadline: 1679309995299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:55,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:55,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:55,306 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:55,306 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:55,306 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:55,306 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:55,314 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:55,314 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:55,321 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:55,322 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:55,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:55,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:55,326 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:55,326 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:55,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:55,326 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:55,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:55,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:55,327 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:55,338 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:55,338 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:55,338 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:55,338 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:55,338 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:55,342 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:55,342 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:55,342 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:55,342 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. started... 2023-03-20 10:58:55,342 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. as already flushing 2023-03-20 10:58:55,342 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:55,342 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:55,342 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:55,342 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:55,342 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:55,343 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:55,350 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:55,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:55,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:55,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:55,351 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:55,353 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:55,353 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:55,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:55,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:55,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:55,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:55,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:55,380 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:55,380 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6545d426[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:55,380 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:55,380 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:55,381 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:55,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:55,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:55,381 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:55,381 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:55,381 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:55,381 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:55,381 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:55,382 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:55,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:55,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:55,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:55,383 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:55,383 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:55,383 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:55,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:55,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:55,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:55,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:55,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:55,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:55,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:55,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:55,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:55,406 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:55,406 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:55,406 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:55,407 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:55,407 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:55,406 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:55,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39896 deadline: 1679309995405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:55,407 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:55,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39904 deadline: 1679309995406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:55,408 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:55,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39798 deadline: 1679309995406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:55,411 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:55,411 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:55,411 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:55,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:55,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:55,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:55,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:55,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:55,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:55,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:55,412 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:55,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:55,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:55,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:55,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:55,414 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:55,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:55,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:55,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:55,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:55,426 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:55,426 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:55,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:55,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:55,426 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:55,426 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:55,426 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:55,426 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:55,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:55,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:55,426 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:55,426 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:55,426 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:55,428 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:55,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:55,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:55,428 DEBUG [Thread-3746] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:55,428 DEBUG [Thread-3746] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:55,430 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:55,530 DEBUG [Thread-3746] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:55,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:55,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:55,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:55,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:55,536 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:55,536 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:55,537 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:55,537 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:55,537 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:55,537 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:55,538 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:55,538 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:55,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:55,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:55,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:55,543 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:55,543 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:55,543 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:55,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:55,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:55,543 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:55,545 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:55,545 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:55,545 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:55,545 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:55,545 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:55,545 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:55,545 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:55,545 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:55,545 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. started... 2023-03-20 10:58:55,545 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. as already flushing 2023-03-20 10:58:55,545 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:55,545 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:55,545 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:55,545 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:55,545 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:55,546 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:55,547 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:55,547 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:55,547 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:55,547 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:55,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:55,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:55,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:55,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:55,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:55,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:55,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:55,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:55,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:55,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26b05353[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:55,562 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:55,562 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:55,570 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:55,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:55,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:55,578 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:55,578 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:55,578 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:55,580 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:55,582 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:55,582 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:55,582 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:55,582 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:55,582 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:55,582 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:55,582 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:55,582 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:55,582 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:55,582 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:55,588 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:55,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:55,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:55,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:55,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:55,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:55,589 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:55,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:55,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:55,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:55,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:55,592 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:55,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:55,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:55,594 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:55,594 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:55,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:55,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:55,598 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:55,598 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:55,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:55,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:55,599 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:55,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:55,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:55,599 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:55,599 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:55,599 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:55,599 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:55,599 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:55,599 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:55,602 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:55,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:55,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:55,610 DEBUG [Thread-3746] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:55,610 DEBUG [Thread-3746] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:55,611 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:55,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39896 deadline: 1679309995609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:55,611 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:55,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39798 deadline: 1679309995609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:55,616 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:55,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39904 deadline: 1679309995612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:55,642 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=137 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/B/f7183f8e8bb9493b9855a9cf2a36f32c 2023-03-20 10:58:55,712 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:55,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39832 deadline: 1679309995708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:55,712 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:55,725 DEBUG [Thread-3738] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4168 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5., hostname=jenkins-hbase20.apache.org,36041,1679309777062, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-20 10:58:55,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39816 deadline: 1679309995710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:55,739 DEBUG [Thread-3740] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4187 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5., hostname=jenkins-hbase20.apache.org,36041,1679309777062, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-20 10:58:55,742 DEBUG [Thread-3746] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:55,748 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=137 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/C/5f09e1bf251e427ba28d11621de7bc6a 2023-03-20 10:58:55,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:55,752 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/A/576b677cd9ad4a5b8cfcc97aea28753a as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/576b677cd9ad4a5b8cfcc97aea28753a 2023-03-20 10:58:55,756 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/576b677cd9ad4a5b8cfcc97aea28753a, entries=350, sequenceid=137, filesize=52.6 K 2023-03-20 10:58:55,757 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/B/f7183f8e8bb9493b9855a9cf2a36f32c as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/f7183f8e8bb9493b9855a9cf2a36f32c 2023-03-20 10:58:55,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:55,772 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/f7183f8e8bb9493b9855a9cf2a36f32c, entries=150, sequenceid=137, filesize=11.8 K 2023-03-20 10:58:55,778 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/C/5f09e1bf251e427ba28d11621de7bc6a as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/5f09e1bf251e427ba28d11621de7bc6a 2023-03-20 10:58:55,791 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/5f09e1bf251e427ba28d11621de7bc6a, entries=150, sequenceid=137, filesize=11.8 K 2023-03-20 10:58:55,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:55,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:55,798 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:55,798 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:55,798 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=139.42 KB/142762 for 96c44321a6ff3c73883afe25fc999ea5 in 689ms, sequenceid=137, compaction requested=true 2023-03-20 10:58:55,798 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:55,798 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:55,798 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 96c44321a6ff3c73883afe25fc999ea5: 2023-03-20 10:58:55,799 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 10:58:55,799 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:55,799 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:55,800 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:58:55,800 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 10:58:55,800 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:55,800 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 105455 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 10:58:55,801 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1912): 96c44321a6ff3c73883afe25fc999ea5/A is initiating minor compaction (all files) 2023-03-20 10:58:55,801 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 96c44321a6ff3c73883afe25fc999ea5/A in TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:55,801 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/d17893d5991a4ae2a44caec61d1385ca, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/56a19abb20c841a18893eed3c7781d1a, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/576b677cd9ad4a5b8cfcc97aea28753a] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp, totalSize=103.0 K 2023-03-20 10:58:55,802 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:55,802 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:55,802 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:55,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:55,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:55,800 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:58:55,803 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:58:55,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:55,806 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting d17893d5991a4ae2a44caec61d1385ca, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1679309931538 2023-03-20 10:58:55,806 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:55,807 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35911 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 10:58:55,807 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): 96c44321a6ff3c73883afe25fc999ea5/B is initiating minor compaction (all files) 2023-03-20 10:58:55,807 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 96c44321a6ff3c73883afe25fc999ea5/B in TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:55,807 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/ac5c7965469f42a7b932280c5c57d2fa, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/8358210835f8404f9d814c6f0867bebe, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/f7183f8e8bb9493b9855a9cf2a36f32c] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp, totalSize=35.1 K 2023-03-20 10:58:55,807 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 56a19abb20c841a18893eed3c7781d1a, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=124, earliestPutTs=1679309932761 2023-03-20 10:58:55,808 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting ac5c7965469f42a7b932280c5c57d2fa, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1679309931538 2023-03-20 10:58:55,808 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 576b677cd9ad4a5b8cfcc97aea28753a, keycount=350, bloomtype=ROW, size=52.6 K, encoding=NONE, compression=NONE, seqNum=137, earliestPutTs=1679309935069 2023-03-20 10:58:55,809 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 8358210835f8404f9d814c6f0867bebe, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=124, earliestPutTs=1679309932761 2023-03-20 10:58:55,810 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting f7183f8e8bb9493b9855a9cf2a36f32c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=137, earliestPutTs=1679309935097 2023-03-20 10:58:55,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:55,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:55,810 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:55,811 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:55,811 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:55,811 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:55,811 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:55,811 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:55,811 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:55,811 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:55,811 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:55,811 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. started... 2023-03-20 10:58:55,811 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 96c44321a6ff3c73883afe25fc999ea5 3/3 column families, dataSize=139.42 KB heapSize=391.34 KB 2023-03-20 10:58:55,812 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 96c44321a6ff3c73883afe25fc999ea5, store=A 2023-03-20 10:58:55,812 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:58:55,812 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 96c44321a6ff3c73883afe25fc999ea5, store=B 2023-03-20 10:58:55,812 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:58:55,812 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 96c44321a6ff3c73883afe25fc999ea5, store=C 2023-03-20 10:58:55,812 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:58:55,899 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] throttle.PressureAwareThroughputController(145): 96c44321a6ff3c73883afe25fc999ea5#A#compaction#520 average throughput is 0.48 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:58:55,912 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): 96c44321a6ff3c73883afe25fc999ea5#B#compaction#522 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:58:55,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 96c44321a6ff3c73883afe25fc999ea5 2023-03-20 10:58:55,920 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. as already flushing 2023-03-20 10:58:55,960 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:55,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39896 deadline: 1679309995945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:55,974 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:55,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39798 deadline: 1679309995960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:55,997 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:55,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39904 deadline: 1679309995976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:56,060 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032031e9312f50504ccd925f32a6bbe1cc37 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032031e9312f50504ccd925f32a6bbe1cc37 2023-03-20 10:58:56,069 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:56,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39896 deadline: 1679309996062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:56,089 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:56,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39798 deadline: 1679309996078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:56,107 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:56,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39904 deadline: 1679309996099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:56,111 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=190, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/A/d3b5109d762340c5a1efa2f613c073ee 2023-03-20 10:58:56,199 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=190 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/B/dc578bc354c3476a886b66bb9a0e6b5b 2023-03-20 10:58:56,265 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=45.49 KB at sequenceid=190 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/C/ffd0cbd4bef0470d86e63124fe0923f8 2023-03-20 10:58:56,271 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/A/d3b5109d762340c5a1efa2f613c073ee as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/d3b5109d762340c5a1efa2f613c073ee 2023-03-20 10:58:56,273 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:56,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39896 deadline: 1679309996272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:56,276 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/d3b5109d762340c5a1efa2f613c073ee, entries=150, sequenceid=190, filesize=25.3 K 2023-03-20 10:58:56,277 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/B/dc578bc354c3476a886b66bb9a0e6b5b as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/dc578bc354c3476a886b66bb9a0e6b5b 2023-03-20 10:58:56,293 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/dc578bc354c3476a886b66bb9a0e6b5b, entries=150, sequenceid=190, filesize=11.8 K 2023-03-20 10:58:56,295 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/C/ffd0cbd4bef0470d86e63124fe0923f8 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/ffd0cbd4bef0470d86e63124fe0923f8 2023-03-20 10:58:56,304 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/ffd0cbd4bef0470d86e63124fe0923f8, entries=150, sequenceid=190, filesize=11.8 K 2023-03-20 10:58:56,305 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~139.42 KB/142762, heapSize ~391.29 KB/400680, currentSize=53.67 KB/54960 for 96c44321a6ff3c73883afe25fc999ea5 in 494ms, sequenceid=190, compaction requested=true 2023-03-20 10:58:56,305 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 96c44321a6ff3c73883afe25fc999ea5: 2023-03-20 10:58:56,305 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:56,305 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:56,305 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:56,305 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:56,305 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:56,305 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:56,312 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:56,312 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:56,312 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:56,314 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:56,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:56,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:56,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:56,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:56,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:56,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:56,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:56,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:56,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:56,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27ed12d0[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:56,317 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:56,317 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:56,319 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:56,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:56,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:56,319 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:56,320 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:56,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 96c44321a6ff3c73883afe25fc999ea5 2023-03-20 10:58:56,320 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 96c44321a6ff3c73883afe25fc999ea5 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-20 10:58:56,321 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 96c44321a6ff3c73883afe25fc999ea5, store=A 2023-03-20 10:58:56,321 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:56,321 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 96c44321a6ff3c73883afe25fc999ea5, store=B 2023-03-20 10:58:56,321 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:56,321 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 96c44321a6ff3c73883afe25fc999ea5, store=C 2023-03-20 10:58:56,321 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:56,326 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:56,326 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:56,326 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:56,327 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:56,327 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:56,328 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:56,328 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:56,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:56,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:56,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:56,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:56,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:56,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:56,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:56,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:56,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:56,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:56,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:56,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:56,331 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:56,331 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:56,331 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:56,331 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:56,331 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:56,337 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:56,341 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:56,342 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:56,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:56,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:56,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:56,342 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:56,342 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:56,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:56,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:56,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:56,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:56,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:56,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:56,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:56,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:56,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:56,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:56,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:56,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:56,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:56,378 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:56,378 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:56,378 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:56,378 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:56,378 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:56,378 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:56,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:56,378 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:56,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:56,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:56,379 DEBUG [Thread-3746] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:56,379 DEBUG [Thread-3746] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:56,378 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:56,379 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:56,379 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:56,379 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:56,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:56,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:56,414 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230320b72d2372c9084bc58a6fc03dd3487477 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320b72d2372c9084bc58a6fc03dd3487477 2023-03-20 10:58:56,437 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/B/a3547e72f05c4630a819691105b83e3b as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/a3547e72f05c4630a819691105b83e3b 2023-03-20 10:58:56,443 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 96c44321a6ff3c73883afe25fc999ea5/B of 96c44321a6ff3c73883afe25fc999ea5 into a3547e72f05c4630a819691105b83e3b(size=12.0 K), total size for store is 23.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:58:56,444 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 96c44321a6ff3c73883afe25fc999ea5: 2023-03-20 10:58:56,444 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5., storeName=96c44321a6ff3c73883afe25fc999ea5/B, priority=13, startTime=1679309935800; duration=0sec 2023-03-20 10:58:56,444 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:58:56,444 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 10:58:56,451 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47947 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 10:58:56,451 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): 96c44321a6ff3c73883afe25fc999ea5/C is initiating minor compaction (all files) 2023-03-20 10:58:56,451 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 96c44321a6ff3c73883afe25fc999ea5/C in TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:56,451 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/349b1558bd3e44d3b8fa0359525c479d, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/89696bd472024b7f90144851f8723817, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/5f09e1bf251e427ba28d11621de7bc6a, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/ffd0cbd4bef0470d86e63124fe0923f8] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp, totalSize=46.8 K 2023-03-20 10:58:56,452 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 349b1558bd3e44d3b8fa0359525c479d, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1679309931538 2023-03-20 10:58:56,452 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 89696bd472024b7f90144851f8723817, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=124, earliestPutTs=1679309932761 2023-03-20 10:58:56,454 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 5f09e1bf251e427ba28d11621de7bc6a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=137, earliestPutTs=1679309935097 2023-03-20 10:58:56,455 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting ffd0cbd4bef0470d86e63124fe0923f8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=190, earliestPutTs=1679309935291 2023-03-20 10:58:56,469 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): 96c44321a6ff3c73883afe25fc999ea5#C#compaction#526 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:58:56,473 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:56,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39798 deadline: 1679309996453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:56,479 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:56,479 DEBUG [Thread-3746] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:56,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39904 deadline: 1679309996458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:56,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:56,481 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:56,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:56,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:56,490 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:56,490 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:56,490 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:56,490 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:56,491 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:56,491 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:56,492 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:56,492 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:56,492 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:56,492 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=203, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/A/6e7dcd9e6c99421fb9ba4903fe52b2e6 2023-03-20 10:58:56,493 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:56,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:56,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:56,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:56,494 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:56,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:56,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:56,495 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:56,510 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:56,510 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:56,510 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:56,510 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:56,510 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:56,534 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:56,534 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:56,534 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:56,534 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. started... 2023-03-20 10:58:56,534 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. as already flushing 2023-03-20 10:58:56,534 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:56,534 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:56,534 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:56,535 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:56,550 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:56,550 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:56,556 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:56,556 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:56,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:56,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:56,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:56,557 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:56,557 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:56,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:56,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:56,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:56,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:56,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:56,559 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:56,559 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@714a378[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:56,559 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:56,559 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:56,560 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:56,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:56,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:56,560 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:56,560 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:56,560 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:56,560 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:56,560 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:56,560 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:56,560 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:56,560 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:56,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:56,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:56,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:56,560 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:56,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:56,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:56,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:56,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:56,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:56,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:56,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:56,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:56,563 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:56,563 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:56,563 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:56,563 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:56,563 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:56,563 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:56,564 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:56,564 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:56,564 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:56,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:56,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:56,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:56,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:56,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:56,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:56,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:56,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:56,565 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:56,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:56,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:56,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:56,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:56,566 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:56,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:56,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:56,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:56,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:56,569 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:56,569 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:56,569 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:56,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:56,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:56,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:56,569 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:56,570 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:56,570 DEBUG [Thread-3746] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:56,570 DEBUG [Thread-3746] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:56,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:56,569 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:56,569 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:56,570 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:56,570 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:56,570 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:56,570 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:56,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:56,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:56,574 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/B/baf414c3fd434b96a71c8fcac54370b6 2023-03-20 10:58:56,576 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:56,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39798 deadline: 1679309996574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:56,578 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:56,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39896 deadline: 1679309996576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:56,586 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:56,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39904 deadline: 1679309996583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:56,610 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/C/5ec49e314c164d7dbaa75f63f02328ae 2023-03-20 10:58:56,642 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/A/6e7dcd9e6c99421fb9ba4903fe52b2e6 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/6e7dcd9e6c99421fb9ba4903fe52b2e6 2023-03-20 10:58:56,647 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/6e7dcd9e6c99421fb9ba4903fe52b2e6, entries=200, sequenceid=203, filesize=32.1 K 2023-03-20 10:58:56,647 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/B/baf414c3fd434b96a71c8fcac54370b6 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/baf414c3fd434b96a71c8fcac54370b6 2023-03-20 10:58:56,650 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/baf414c3fd434b96a71c8fcac54370b6, entries=150, sequenceid=203, filesize=11.8 K 2023-03-20 10:58:56,651 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/C/5ec49e314c164d7dbaa75f63f02328ae as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/5ec49e314c164d7dbaa75f63f02328ae 2023-03-20 10:58:56,664 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/5ec49e314c164d7dbaa75f63f02328ae, entries=150, sequenceid=203, filesize=11.8 K 2023-03-20 10:58:56,664 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 96c44321a6ff3c73883afe25fc999ea5 in 344ms, sequenceid=203, compaction requested=true 2023-03-20 10:58:56,664 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 96c44321a6ff3c73883afe25fc999ea5: 2023-03-20 10:58:56,665 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:58:56,665 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-20 10:58:56,665 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-03-20 10:58:56,670 DEBUG [Thread-3746] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:56,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:56,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:56,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:56,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:56,691 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:56,691 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:56,691 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:56,691 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:56,691 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:56,691 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:56,692 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:56,692 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:56,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:56,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:56,692 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:56,692 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:56,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:56,692 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:56,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:56,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:56,693 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:56,702 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:56,702 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:56,702 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:56,702 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:56,702 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:56,706 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:56,706 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:56,706 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:56,706 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. started... 2023-03-20 10:58:56,706 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 96c44321a6ff3c73883afe25fc999ea5 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-20 10:58:56,706 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 96c44321a6ff3c73883afe25fc999ea5, store=A 2023-03-20 10:58:56,707 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:56,707 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 96c44321a6ff3c73883afe25fc999ea5, store=B 2023-03-20 10:58:56,707 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:56,707 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 96c44321a6ff3c73883afe25fc999ea5, store=C 2023-03-20 10:58:56,707 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:56,725 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303206da75ce969684a6b8cdaa5f79aa88149 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303206da75ce969684a6b8cdaa5f79aa88149 2023-03-20 10:58:56,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 96c44321a6ff3c73883afe25fc999ea5 2023-03-20 10:58:56,778 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. as already flushing 2023-03-20 10:58:56,813 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:56,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39798 deadline: 1679309996807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:56,815 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:56,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39904 deadline: 1679309996813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:56,889 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/C/d6ad15fa78c64850abe8bb71ab916eb3 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/d6ad15fa78c64850abe8bb71ab916eb3 2023-03-20 10:58:56,905 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 96c44321a6ff3c73883afe25fc999ea5/C of 96c44321a6ff3c73883afe25fc999ea5 into d6ad15fa78c64850abe8bb71ab916eb3(size=12.0 K), total size for store is 23.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:58:56,905 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 96c44321a6ff3c73883afe25fc999ea5: 2023-03-20 10:58:56,905 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5., storeName=96c44321a6ff3c73883afe25fc999ea5/C, priority=12, startTime=1679309935803; duration=0sec 2023-03-20 10:58:56,905 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-03-20 10:58:56,905 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 3 compacting, 2 eligible, 16 blocking 2023-03-20 10:58:56,909 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-20 10:58:56,909 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-20 10:58:56,909 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. because compaction request was cancelled 2023-03-20 10:58:56,909 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 10:58:56,913 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 10:58:56,913 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): 96c44321a6ff3c73883afe25fc999ea5/B is initiating minor compaction (all files) 2023-03-20 10:58:56,913 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 96c44321a6ff3c73883afe25fc999ea5/B in TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:56,913 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/a3547e72f05c4630a819691105b83e3b, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/dc578bc354c3476a886b66bb9a0e6b5b, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/baf414c3fd434b96a71c8fcac54370b6] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp, totalSize=35.5 K 2023-03-20 10:58:56,915 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting a3547e72f05c4630a819691105b83e3b, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=137, earliestPutTs=1679309935097 2023-03-20 10:58:56,916 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting dc578bc354c3476a886b66bb9a0e6b5b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=190, earliestPutTs=1679309935291 2023-03-20 10:58:56,916 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting baf414c3fd434b96a71c8fcac54370b6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1679309935917 2023-03-20 10:58:56,918 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:56,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39798 deadline: 1679309996916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:56,919 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:56,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39904 deadline: 1679309996916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:56,923 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): 96c44321a6ff3c73883afe25fc999ea5#B#compaction#530 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:58:56,985 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/A/7159cdee5fa44dadae506246e1138b1b as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/7159cdee5fa44dadae506246e1138b1b 2023-03-20 10:58:57,014 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 96c44321a6ff3c73883afe25fc999ea5/A of 96c44321a6ff3c73883afe25fc999ea5 into 7159cdee5fa44dadae506246e1138b1b(size=25.5 K), total size for store is 82.9 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-03-20 10:58:57,014 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 96c44321a6ff3c73883afe25fc999ea5: 2023-03-20 10:58:57,014 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5., storeName=96c44321a6ff3c73883afe25fc999ea5/A, priority=13, startTime=1679309935799; duration=1sec 2023-03-20 10:58:57,014 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:58:57,014 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-20 10:58:57,018 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-20 10:58:57,018 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-20 10:58:57,019 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. because compaction request was cancelled 2023-03-20 10:58:57,084 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:57,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39896 deadline: 1679309997083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:57,122 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:57,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39904 deadline: 1679309997120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:57,122 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:57,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39798 deadline: 1679309997121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:57,166 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=223, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/A/e7304005dee84e6ca4f434b375db2f57 2023-03-20 10:58:57,338 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/B/e73c6c855a134523855c18812a8cfd88 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/e73c6c855a134523855c18812a8cfd88 2023-03-20 10:58:57,345 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 96c44321a6ff3c73883afe25fc999ea5/B of 96c44321a6ff3c73883afe25fc999ea5 into e73c6c855a134523855c18812a8cfd88(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:58:57,346 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 96c44321a6ff3c73883afe25fc999ea5: 2023-03-20 10:58:57,346 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5., storeName=96c44321a6ff3c73883afe25fc999ea5/B, priority=13, startTime=1679309936665; duration=0sec 2023-03-20 10:58:57,346 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:58:57,425 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:57,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39798 deadline: 1679309997423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:57,427 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:57,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39904 deadline: 1679309997425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:57,622 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=223 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/B/e8b702252a014fb0a0030659ae17e1c1 2023-03-20 10:58:57,931 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:57,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39798 deadline: 1679309997929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:57,940 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:57,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39904 deadline: 1679309997938, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:58,081 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=223 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/C/43bbc546850743e88c90457a64cbc411 2023-03-20 10:58:58,093 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:58,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39896 deadline: 1679309998088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:58,118 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/A/e7304005dee84e6ca4f434b375db2f57 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/e7304005dee84e6ca4f434b375db2f57 2023-03-20 10:58:58,135 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/e7304005dee84e6ca4f434b375db2f57, entries=150, sequenceid=223, filesize=25.3 K 2023-03-20 10:58:58,138 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/B/e8b702252a014fb0a0030659ae17e1c1 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/e8b702252a014fb0a0030659ae17e1c1 2023-03-20 10:58:58,149 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/e8b702252a014fb0a0030659ae17e1c1, entries=150, sequenceid=223, filesize=11.8 K 2023-03-20 10:58:58,149 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/C/43bbc546850743e88c90457a64cbc411 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/43bbc546850743e88c90457a64cbc411 2023-03-20 10:58:58,153 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/43bbc546850743e88c90457a64cbc411, entries=150, sequenceid=223, filesize=11.8 K 2023-03-20 10:58:58,153 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 96c44321a6ff3c73883afe25fc999ea5 in 1447ms, sequenceid=223, compaction requested=true 2023-03-20 10:58:58,153 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 96c44321a6ff3c73883afe25fc999ea5: 2023-03-20 10:58:58,153 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:58,153 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:58,153 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:58,153 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:58,154 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:58,154 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:58,154 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:58,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:58,154 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:58,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:58,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:58,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:58,155 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:58,155 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:58,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:58,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:58,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:58,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:58,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:58,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4cbbb358[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:58,156 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:58,157 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:58,157 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:58,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:58,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:58,157 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:58,157 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:58,157 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:58,157 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:58,158 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:58,158 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:58,158 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:58,158 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:58,158 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:58,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:58,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:58,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:58,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:58,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:58,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:58,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:58,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:58,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:58,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:58,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:58,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:58,161 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:58,161 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:58,161 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:58,161 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:58,161 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:58,162 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:58,162 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:58,162 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:58,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:58,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:58,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:58,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:58,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:58,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:58,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:58,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:58,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:58,163 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:58,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:58,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:58,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:58,164 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:58,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:58,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:58,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:58,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:58,169 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:58,169 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:58,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:58,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:58,169 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:58,169 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:58,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:58,170 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:58,170 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:58,170 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:58,170 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:58,170 DEBUG [Thread-3746] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:58,170 DEBUG [Thread-3746] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:58,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:58,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:58,170 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:58,174 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:58,174 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:58,174 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:58,274 DEBUG [Thread-3746] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:58,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:58,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:58,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:58,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:58,306 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:58,306 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:58,306 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:58,306 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:58,307 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:58,307 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:58,308 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:58,309 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:58,309 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:58,310 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:58,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:58,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:58,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:58,311 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:58,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:58,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:58,311 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:58,320 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:58,320 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:58,320 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:58,320 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:58,320 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:58,330 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:58,330 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:58,333 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:58,333 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. started... 2023-03-20 10:58:58,333 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 96c44321a6ff3c73883afe25fc999ea5 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-20 10:58:58,333 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 96c44321a6ff3c73883afe25fc999ea5, store=A 2023-03-20 10:58:58,334 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:58,334 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 96c44321a6ff3c73883afe25fc999ea5, store=B 2023-03-20 10:58:58,334 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:58,334 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 96c44321a6ff3c73883afe25fc999ea5, store=C 2023-03-20 10:58:58,334 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:58,394 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230320cefc6a6e7a9e42c0a10fb0d03809cf43 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320cefc6a6e7a9e42c0a10fb0d03809cf43 2023-03-20 10:58:58,817 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=239, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/A/737085474471476bba636cc9ff3f15d9 2023-03-20 10:58:58,866 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=239 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/B/78dd247c21d7478eb666bc71a20cdf05 2023-03-20 10:58:58,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 96c44321a6ff3c73883afe25fc999ea5 2023-03-20 10:58:58,963 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. as already flushing 2023-03-20 10:58:59,075 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:59,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39798 deadline: 1679309999061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:59,076 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:59,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39904 deadline: 1679309999061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:59,185 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:59,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39904 deadline: 1679309999183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:59,185 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:59,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39798 deadline: 1679309999183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:59,331 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=239 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/C/639059aa57ab48fe8037cc95786413b4 2023-03-20 10:58:59,336 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/A/737085474471476bba636cc9ff3f15d9 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/737085474471476bba636cc9ff3f15d9 2023-03-20 10:58:59,341 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/737085474471476bba636cc9ff3f15d9, entries=150, sequenceid=239, filesize=25.3 K 2023-03-20 10:58:59,342 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/B/78dd247c21d7478eb666bc71a20cdf05 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/78dd247c21d7478eb666bc71a20cdf05 2023-03-20 10:58:59,347 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/78dd247c21d7478eb666bc71a20cdf05, entries=150, sequenceid=239, filesize=11.8 K 2023-03-20 10:58:59,347 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/C/639059aa57ab48fe8037cc95786413b4 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/639059aa57ab48fe8037cc95786413b4 2023-03-20 10:58:59,351 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/639059aa57ab48fe8037cc95786413b4, entries=150, sequenceid=239, filesize=11.8 K 2023-03-20 10:58:59,352 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 96c44321a6ff3c73883afe25fc999ea5 in 1019ms, sequenceid=239, compaction requested=true 2023-03-20 10:58:59,352 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 96c44321a6ff3c73883afe25fc999ea5: 2023-03-20 10:58:59,352 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:59,352 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:59,352 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:59,352 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:59,352 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:59,353 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:59,353 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:59,353 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:59,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:59,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:59,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:59,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:59,354 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:59,354 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:59,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:59,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:59,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:59,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:59,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:59,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68b54b57[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:59,355 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:59,355 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:59,356 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:59,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:59,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:59,356 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:59,356 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:59,356 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:59,357 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:59,357 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:59,357 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:59,361 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:59,361 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:59,361 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:59,361 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:59,361 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:59,361 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:59,361 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:59,361 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:59,363 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:59,363 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:59,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:59,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:59,363 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:59,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:59,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:59,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:59,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:59,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:59,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:59,365 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:59,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:59,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:59,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:59,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:59,366 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:59,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:59,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:59,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:59,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:59,369 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:59,369 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:59,369 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:59,369 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:59,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:59,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:59,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:59,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:59,369 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:59,369 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:59,369 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:59,369 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:59,369 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:59,369 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:59,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:59,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:59,370 DEBUG [Thread-3746] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:59,370 DEBUG [Thread-3746] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:59,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 96c44321a6ff3c73883afe25fc999ea5 2023-03-20 10:58:59,390 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 96c44321a6ff3c73883afe25fc999ea5 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-20 10:58:59,390 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 96c44321a6ff3c73883afe25fc999ea5, store=A 2023-03-20 10:58:59,390 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:59,390 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 96c44321a6ff3c73883afe25fc999ea5, store=B 2023-03-20 10:58:59,391 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:59,391 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 96c44321a6ff3c73883afe25fc999ea5, store=C 2023-03-20 10:58:59,391 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:58:59,411 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230320af1aef6144754525aa70331acdd5a5a4 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320af1aef6144754525aa70331acdd5a5a4 2023-03-20 10:58:59,414 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:59,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39798 deadline: 1679309999412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:59,416 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:59,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39904 deadline: 1679309999414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:59,470 DEBUG [Thread-3746] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:59,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:59,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:59,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:59,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:59,474 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:59,474 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:59,474 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:59,474 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:59,475 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:59,475 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:59,475 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:59,475 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:59,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:59,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:59,476 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:59,476 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:59,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:59,477 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:59,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:59,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:59,477 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:59,477 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:59,477 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:59,477 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:59,477 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:59,477 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:59,482 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:59,482 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:59,482 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:59,482 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. started... 2023-03-20 10:58:59,482 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. as already flushing 2023-03-20 10:58:59,482 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:59,482 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:59,482 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:59,482 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:59,482 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:59,482 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:59,486 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:59,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:59,486 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:59,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:59,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:59,486 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:59,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:59,486 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:59,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:59,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:59,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:59,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:59,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:59,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2781bf03[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:59,488 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:59,488 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:59,489 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:59,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:59,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:59,490 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:59,490 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:59,490 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:59,490 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:59,490 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:59,491 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:59,491 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:59,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:59,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:59,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:59,491 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:59,491 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:59,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:59,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:59,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:59,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:59,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:59,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:59,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:59,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:59,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:59,496 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:59,496 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:59,496 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:59,496 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:59,496 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:59,503 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:59,503 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:59,503 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:59,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:59,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:59,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:59,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:59,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:59,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:59,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:59,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:59,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:59,507 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:59,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:59,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:59,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:59,509 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:59,511 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:59,512 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:59,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:59,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:59,512 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:59,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:59,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:59,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:59,512 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:59,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:59,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:59,512 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:59,512 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:59,512 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:59,513 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:59,513 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:59,513 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:59,513 DEBUG [Thread-3746] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:59,513 DEBUG [Thread-3746] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:59,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:59,515 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:59,519 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:59,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39798 deadline: 1679309999516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:59,519 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:59,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39904 deadline: 1679309999517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:59,613 DEBUG [Thread-3746] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:59,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:59,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:59,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:59,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:59,628 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:59,628 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:59,628 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:59,628 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:59,629 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:59,629 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:59,630 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:59,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:59,630 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:59,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:59,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:59,631 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:59,631 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:59,631 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:59,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:59,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:59,631 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:59,632 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:58:59,632 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:59,632 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:59,632 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:59,632 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:59,632 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:59,632 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:59,632 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:59,632 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. started... 2023-03-20 10:58:59,632 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. as already flushing 2023-03-20 10:58:59,632 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:59,633 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:59,633 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:59,633 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:59,633 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:59,633 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:59,633 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:59,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:59,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:59,634 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:59,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:59,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:59,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:59,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:59,635 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:59,635 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:59,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:59,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:59,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:59,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5acc7e8[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:59,636 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:59,636 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:59,636 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:59,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:59,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:59,637 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:59,637 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:59,637 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:59,637 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:59,637 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:59,637 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:59,637 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:59,638 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:59,638 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:59,638 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:59,638 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:59,638 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:59,638 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:59,638 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:59,640 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:59,640 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:59,640 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:59,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:59,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:59,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:59,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:59,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:59,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:59,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:59,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:59,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:59,641 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:59,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:59,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:59,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:59,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:59,647 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:59,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:59,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:59,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:59,666 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:59,666 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:59,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:59,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:59,666 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:59,666 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:59,666 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:59,666 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:59,666 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:59,666 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:59,666 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:59,666 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:59,666 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:59,666 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:59,666 DEBUG [Thread-3746] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:59,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:59,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:59,667 DEBUG [Thread-3746] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:59,684 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:58:59,723 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:59,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39798 deadline: 1679309999720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:59,735 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:59,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39904 deadline: 1679309999733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:59,759 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:59,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39832 deadline: 1679309999758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:59,761 DEBUG [Thread-3738] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8215 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5., hostname=jenkins-hbase20.apache.org,36041,1679309777062, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-20 10:58:59,767 DEBUG [Thread-3746] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:59,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:59,769 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:59,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:59,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:59,776 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:59,777 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:59,777 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:59,777 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:59,777 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:58:59,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39816 deadline: 1679309999775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:59,779 DEBUG [Thread-3740] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8234 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5., hostname=jenkins-hbase20.apache.org,36041,1679309777062, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-20 10:58:59,786 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:59,786 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:59,793 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:59,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:59,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:59,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:59,794 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:59,798 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:59,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:59,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:59,798 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:58:59,801 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:58:59,806 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:58:59,806 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:59,806 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:58:59,810 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:58:59,810 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:58:59,813 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:59,813 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. started... 2023-03-20 10:58:59,813 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. as already flushing 2023-03-20 10:58:59,813 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:59,813 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:58:59,813 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:58:59,813 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:59,813 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:58:59,813 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:58:59,814 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:59,814 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:59,819 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:59,820 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:59,821 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:58:59,821 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:59,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:59,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:59,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:59,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:59,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:59,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:59,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:59,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:59,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:58:59,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@149c8576[Count = 0] remaining members to acquire global barrier 2023-03-20 10:58:59,834 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:58:59,834 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:59,836 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:59,836 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:58:59,838 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:59,838 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=259, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/A/2ff805f7b93b4c0b9a10bf3e40dd789d 2023-03-20 10:58:59,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:59,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:59,840 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:58:59,840 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:58:59,840 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:58:59,844 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:59,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:59,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:59,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:59,844 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:58:59,844 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:59,844 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:58:59,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:59,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:59,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:59,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:59,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:59,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:59,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:59,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:58:59,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:58:59,848 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:58:59,848 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:58:59,848 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:58:59,848 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:58:59,848 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:58:59,852 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:59,852 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:59,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:59,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:59,852 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:59,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:59,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:58:59,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:58:59,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:59,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:59,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:58:59,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:59,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:59,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:58:59,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:59,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:59,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:58:59,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:58:59,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:59,858 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:59,859 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:59,868 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=259 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/B/5ce25c323bad4bf3afe9cef379845c3f 2023-03-20 10:58:59,869 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:59,869 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:59,869 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:58:59,869 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:59,869 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:59,869 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:59,869 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:58:59,869 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:59,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:58:59,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:58:59,870 DEBUG [Thread-3746] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:58:59,870 DEBUG [Thread-3746] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:58:59,870 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:59,870 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:58:59,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:59,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:59,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:58:59,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:58:59,890 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=259 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/C/5cf3d45f8d6f4612b67f81f25e543d64 2023-03-20 10:58:59,895 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/A/2ff805f7b93b4c0b9a10bf3e40dd789d as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/2ff805f7b93b4c0b9a10bf3e40dd789d 2023-03-20 10:58:59,899 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/2ff805f7b93b4c0b9a10bf3e40dd789d, entries=200, sequenceid=259, filesize=32.2 K 2023-03-20 10:58:59,900 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/B/5ce25c323bad4bf3afe9cef379845c3f as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/5ce25c323bad4bf3afe9cef379845c3f 2023-03-20 10:58:59,905 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/5ce25c323bad4bf3afe9cef379845c3f, entries=150, sequenceid=259, filesize=11.9 K 2023-03-20 10:58:59,906 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/C/5cf3d45f8d6f4612b67f81f25e543d64 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/5cf3d45f8d6f4612b67f81f25e543d64 2023-03-20 10:58:59,918 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/5cf3d45f8d6f4612b67f81f25e543d64, entries=150, sequenceid=259, filesize=11.9 K 2023-03-20 10:58:59,919 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 96c44321a6ff3c73883afe25fc999ea5 in 529ms, sequenceid=259, compaction requested=true 2023-03-20 10:58:59,919 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 96c44321a6ff3c73883afe25fc999ea5: 2023-03-20 10:58:59,919 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:58:59,919 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-20 10:58:59,919 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 10:58:59,919 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:58:59,920 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:58:59,921 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48552 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 10:58:59,921 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): 96c44321a6ff3c73883afe25fc999ea5/B is initiating minor compaction (all files) 2023-03-20 10:58:59,921 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 96c44321a6ff3c73883afe25fc999ea5/B in TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:59,921 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/e73c6c855a134523855c18812a8cfd88, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/e8b702252a014fb0a0030659ae17e1c1, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/78dd247c21d7478eb666bc71a20cdf05, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/5ce25c323bad4bf3afe9cef379845c3f] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp, totalSize=47.4 K 2023-03-20 10:58:59,922 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting e73c6c855a134523855c18812a8cfd88, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1679309935917 2023-03-20 10:58:59,922 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting e8b702252a014fb0a0030659ae17e1c1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=223, earliestPutTs=1679309936435 2023-03-20 10:58:59,932 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 169662 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-20 10:58:59,932 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1912): 96c44321a6ff3c73883afe25fc999ea5/A is initiating minor compaction (all files) 2023-03-20 10:58:59,932 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 96c44321a6ff3c73883afe25fc999ea5/A in TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:58:59,932 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/7159cdee5fa44dadae506246e1138b1b, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/d3b5109d762340c5a1efa2f613c073ee, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/6e7dcd9e6c99421fb9ba4903fe52b2e6, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/e7304005dee84e6ca4f434b375db2f57, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/737085474471476bba636cc9ff3f15d9, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/2ff805f7b93b4c0b9a10bf3e40dd789d] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp, totalSize=165.7 K 2023-03-20 10:58:59,933 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 78dd247c21d7478eb666bc71a20cdf05, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=239, earliestPutTs=1679309936786 2023-03-20 10:58:59,933 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 7159cdee5fa44dadae506246e1138b1b, keycount=150, bloomtype=ROW, size=25.5 K, encoding=NONE, compression=NONE, seqNum=137, earliestPutTs=1679309935097 2023-03-20 10:58:59,933 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 5ce25c323bad4bf3afe9cef379845c3f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=259, earliestPutTs=1679309939055 2023-03-20 10:58:59,933 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting d3b5109d762340c5a1efa2f613c073ee, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=190, earliestPutTs=1679309935291 2023-03-20 10:58:59,939 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 6e7dcd9e6c99421fb9ba4903fe52b2e6, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1679309935917 2023-03-20 10:58:59,946 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting e7304005dee84e6ca4f434b375db2f57, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=223, earliestPutTs=1679309936435 2023-03-20 10:58:59,947 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 737085474471476bba636cc9ff3f15d9, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=239, earliestPutTs=1679309936786 2023-03-20 10:58:59,947 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 2ff805f7b93b4c0b9a10bf3e40dd789d, keycount=200, bloomtype=ROW, size=32.2 K, encoding=NONE, compression=NONE, seqNum=259, earliestPutTs=1679309939038 2023-03-20 10:58:59,960 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): 96c44321a6ff3c73883afe25fc999ea5#B#compaction#539 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:58:59,970 DEBUG [Thread-3746] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:58:59,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:58:59,976 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:58:59,981 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] throttle.PressureAwareThroughputController(145): 96c44321a6ff3c73883afe25fc999ea5#A#compaction#540 average throughput is 1.79 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:58:59,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:58:59,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:58:59,982 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:58:59,982 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:58:59,982 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:58:59,982 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:58:59,983 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:59,983 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:59,984 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:59,984 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:58:59,984 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:58:59,984 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:58:59,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:58:59,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:58:59,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:59,985 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:58:59,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:58:59,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:58:59,985 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:59:00,002 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:59:00,002 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:59:00,002 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:59:00,002 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:59:00,002 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:59:00,018 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:59:00,018 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:59:00,019 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:59:00,019 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. started... 2023-03-20 10:59:00,020 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 96c44321a6ff3c73883afe25fc999ea5 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-20 10:59:00,021 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 96c44321a6ff3c73883afe25fc999ea5, store=A 2023-03-20 10:59:00,021 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:59:00,021 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 96c44321a6ff3c73883afe25fc999ea5, store=B 2023-03-20 10:59:00,021 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:59:00,021 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 96c44321a6ff3c73883afe25fc999ea5, store=C 2023-03-20 10:59:00,021 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:59:00,027 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. as already flushing 2023-03-20 10:59:00,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 96c44321a6ff3c73883afe25fc999ea5 2023-03-20 10:59:00,046 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230320c5446a772d384e659268ef93b3aa47de to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320c5446a772d384e659268ef93b3aa47de 2023-03-20 10:59:00,056 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=271, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/A/fda06680636e4a7680027a86919db163 2023-03-20 10:59:00,087 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:00,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39904 deadline: 1679310000084, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:00,088 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:00,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39798 deadline: 1679310000084, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:00,089 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=271 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/B/18918dd42766411f90a8b8fd50500f3c 2023-03-20 10:59:00,104 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:00,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39896 deadline: 1679310000102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:00,106 DEBUG [Thread-3742] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4160 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5., hostname=jenkins-hbase20.apache.org,36041,1679309777062, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-20 10:59:00,191 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:00,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39904 deadline: 1679310000189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:00,191 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:00,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39798 deadline: 1679310000189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:00,385 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/B/653e7e82f9e148b3b0c275e5de0d5427 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/653e7e82f9e148b3b0c275e5de0d5427 2023-03-20 10:59:00,395 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:00,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39904 deadline: 1679310000393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:00,400 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 96c44321a6ff3c73883afe25fc999ea5/B of 96c44321a6ff3c73883afe25fc999ea5 into 653e7e82f9e148b3b0c275e5de0d5427(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:59:00,400 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 96c44321a6ff3c73883afe25fc999ea5: 2023-03-20 10:59:00,400 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5., storeName=96c44321a6ff3c73883afe25fc999ea5/B, priority=12, startTime=1679309939919; duration=0sec 2023-03-20 10:59:00,400 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:59:00,400 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-20 10:59:00,401 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:00,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39798 deadline: 1679310000398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:00,405 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60520 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-20 10:59:00,405 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): 96c44321a6ff3c73883afe25fc999ea5/C is initiating minor compaction (all files) 2023-03-20 10:59:00,405 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 96c44321a6ff3c73883afe25fc999ea5/C in TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:59:00,406 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/d6ad15fa78c64850abe8bb71ab916eb3, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/5ec49e314c164d7dbaa75f63f02328ae, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/43bbc546850743e88c90457a64cbc411, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/639059aa57ab48fe8037cc95786413b4, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/5cf3d45f8d6f4612b67f81f25e543d64] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp, totalSize=59.1 K 2023-03-20 10:59:00,406 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting d6ad15fa78c64850abe8bb71ab916eb3, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=190, earliestPutTs=1679309935291 2023-03-20 10:59:00,406 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 5ec49e314c164d7dbaa75f63f02328ae, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1679309935917 2023-03-20 10:59:00,407 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 43bbc546850743e88c90457a64cbc411, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=223, earliestPutTs=1679309936435 2023-03-20 10:59:00,407 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 639059aa57ab48fe8037cc95786413b4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=239, earliestPutTs=1679309936786 2023-03-20 10:59:00,407 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 5cf3d45f8d6f4612b67f81f25e543d64, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=259, earliestPutTs=1679309939055 2023-03-20 10:59:00,420 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): 96c44321a6ff3c73883afe25fc999ea5#C#compaction#544 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:59:00,514 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=271 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/C/a01c3fdcd28c48d9855ba61d4957b4cd 2023-03-20 10:59:00,523 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/A/fda06680636e4a7680027a86919db163 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/fda06680636e4a7680027a86919db163 2023-03-20 10:59:00,527 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/fda06680636e4a7680027a86919db163, entries=150, sequenceid=271, filesize=25.4 K 2023-03-20 10:59:00,528 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/B/18918dd42766411f90a8b8fd50500f3c as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/18918dd42766411f90a8b8fd50500f3c 2023-03-20 10:59:00,539 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/18918dd42766411f90a8b8fd50500f3c, entries=150, sequenceid=271, filesize=11.9 K 2023-03-20 10:59:00,546 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/C/a01c3fdcd28c48d9855ba61d4957b4cd as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/a01c3fdcd28c48d9855ba61d4957b4cd 2023-03-20 10:59:00,563 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/a01c3fdcd28c48d9855ba61d4957b4cd, entries=150, sequenceid=271, filesize=11.9 K 2023-03-20 10:59:00,564 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 96c44321a6ff3c73883afe25fc999ea5 in 544ms, sequenceid=271, compaction requested=false 2023-03-20 10:59:00,564 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 96c44321a6ff3c73883afe25fc999ea5: 2023-03-20 10:59:00,564 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:59:00,564 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:59:00,564 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:59:00,564 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:00,564 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:59:00,564 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:59:00,570 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:00,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:00,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:59:00,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:59:00,571 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:00,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:59:00,573 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:00,573 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:59:00,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:59:00,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:00,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:00,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:59:00,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:59:00,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56318b41[Count = 0] remaining members to acquire global barrier 2023-03-20 10:59:00,580 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:59:00,580 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:00,626 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:00,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:00,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:00,626 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:00,626 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:59:00,630 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:59:00,630 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:59:00,630 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:59:00,647 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:00,647 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:59:00,647 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:59:00,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:00,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:59:00,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:59:00,647 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:59:00,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:59:00,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:59:00,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:00,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:00,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:59:00,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:00,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:00,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:59:00,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:59:00,691 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:59:00,691 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:59:00,691 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:59:00,691 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:59:00,691 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:59:00,693 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:00,693 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:00,694 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:59:00,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:00,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:00,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:00,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:59:00,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:59:00,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:59:00,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:59:00,695 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:00,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:59:00,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:00,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:00,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:59:00,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:00,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:00,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:59:00,697 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:00,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:00,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:00,700 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 96c44321a6ff3c73883afe25fc999ea5 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-03-20 10:59:00,700 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 96c44321a6ff3c73883afe25fc999ea5, store=A 2023-03-20 10:59:00,700 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:59:00,700 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 96c44321a6ff3c73883afe25fc999ea5, store=B 2023-03-20 10:59:00,700 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:59:00,700 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 96c44321a6ff3c73883afe25fc999ea5, store=C 2023-03-20 10:59:00,700 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:59:00,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 96c44321a6ff3c73883afe25fc999ea5 2023-03-20 10:59:00,704 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:59:00,704 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:59:00,704 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:59:00,704 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:00,704 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:00,704 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:00,704 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:00,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:59:00,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:59:00,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:59:00,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:59:00,704 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:00,704 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:00,704 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:00,704 DEBUG [Thread-3746] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:59:00,704 DEBUG [Thread-3746] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:59:00,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:59:00,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:59:00,739 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:00,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39904 deadline: 1679310000735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:00,740 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:00,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39798 deadline: 1679310000736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:00,756 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032089077b5d20ad4d82a65ce62763212b17 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032089077b5d20ad4d82a65ce62763212b17 2023-03-20 10:59:00,805 DEBUG [Thread-3746] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:59:00,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:59:00,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:59:00,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:59:00,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:59:00,813 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:59:00,814 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:59:00,814 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:59:00,814 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:59:00,815 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:00,815 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:00,817 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:59:00,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:59:00,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:59:00,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:00,818 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:00,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:59:00,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:00,822 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:59:00,823 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:00,824 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:59:00,824 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:59:00,826 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:59:00,826 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:59:00,826 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:59:00,826 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:59:00,826 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:59:00,826 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:00,826 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:59:00,826 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:59:00,829 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. started... 2023-03-20 10:59:00,829 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. as already flushing 2023-03-20 10:59:00,829 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:59:00,830 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:59:00,830 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:59:00,830 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:00,830 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:59:00,830 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:59:00,836 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:00,836 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:00,836 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:00,836 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:59:00,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:00,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:59:00,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:59:00,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:59:00,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:59:00,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:00,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:00,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:59:00,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:59:00,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ce24f23[Count = 0] remaining members to acquire global barrier 2023-03-20 10:59:00,846 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:59:00,846 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:00,847 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:00,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39798 deadline: 1679310000843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:00,848 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:00,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39904 deadline: 1679310000845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:00,852 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:00,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:00,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:00,852 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:59:00,852 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:59:00,852 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:00,853 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:59:00,852 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:59:00,853 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:59:00,853 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:59:00,853 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:59:00,853 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:00,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:00,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:59:00,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:59:00,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:59:00,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:59:00,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:00,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:00,865 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/C/3fbf0c8c0d324c47a7daa80b2a593e8d as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/3fbf0c8c0d324c47a7daa80b2a593e8d 2023-03-20 10:59:00,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:59:00,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:00,886 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 96c44321a6ff3c73883afe25fc999ea5/C of 96c44321a6ff3c73883afe25fc999ea5 into 3fbf0c8c0d324c47a7daa80b2a593e8d(size=12.3 K), total size for store is 24.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:59:00,887 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 96c44321a6ff3c73883afe25fc999ea5: 2023-03-20 10:59:00,887 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5., storeName=96c44321a6ff3c73883afe25fc999ea5/C, priority=11, startTime=1679309939920; duration=0sec 2023-03-20 10:59:00,887 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:59:00,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:00,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:59:00,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:59:00,891 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:59:00,891 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:59:00,891 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:59:00,891 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:59:00,891 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:59:00,892 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:00,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:00,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:59:00,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:59:00,892 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:00,892 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:59:00,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:00,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:00,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:59:00,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:59:00,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:59:00,893 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:00,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:00,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:00,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:59:00,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:00,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:00,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:59:00,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:00,895 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:00,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:00,896 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:59:00,896 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:59:00,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:59:00,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:59:00,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:59:00,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:59:00,897 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:00,897 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:00,897 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:00,897 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:00,897 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:00,897 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:00,897 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:00,897 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:59:00,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:59:00,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:59:00,898 DEBUG [Thread-3746] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:59:00,898 DEBUG [Thread-3746] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:59:00,998 DEBUG [Thread-3746] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:59:00,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:59:01,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:59:01,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:59:01,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:59:01,026 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:59:01,026 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:59:01,026 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:59:01,026 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:59:01,027 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:01,027 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:01,028 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:59:01,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:59:01,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:59:01,028 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:01,029 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:01,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:59:01,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:01,029 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:59:01,030 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,030 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:59:01,030 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:59:01,031 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:59:01,031 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:59:01,031 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:59:01,031 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:59:01,050 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:01,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39904 deadline: 1679310001049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,056 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:01,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39798 deadline: 1679310001050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,058 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:59:01,058 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:59:01,058 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:59:01,058 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. started... 2023-03-20 10:59:01,058 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. as already flushing 2023-03-20 10:59:01,058 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:59:01,058 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:59:01,058 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:59:01,058 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,059 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:59:01,059 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:59:01,067 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:01,067 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:01,067 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:59:01,068 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:59:01,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:59:01,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:59:01,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:59:01,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:01,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:59:01,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:59:01,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f9254d0[Count = 0] remaining members to acquire global barrier 2023-03-20 10:59:01,086 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:59:01,086 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:01,090 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:01,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:01,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:01,090 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:59:01,090 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:59:01,090 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:59:01,092 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:59:01,092 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:59:01,092 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:59:01,093 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,093 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:59:01,093 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:59:01,093 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:59:01,093 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:59:01,093 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:59:01,093 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:59:01,093 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:59:01,103 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:01,104 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:59:01,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:01,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:01,105 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:01,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:01,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:59:01,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:59:01,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:59:01,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:59:01,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:01,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:59:01,111 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:01,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:59:01,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:01,114 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:59:01,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:01,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,132 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:59:01,132 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:59:01,133 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,133 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:01,133 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:01,133 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,133 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:01,133 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:01,133 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:01,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:59:01,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:59:01,133 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:59:01,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:59:01,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:59:01,133 DEBUG [Thread-3746] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:59:01,133 DEBUG [Thread-3746] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:59:01,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:59:01,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:59:01,181 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=293, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/A/9cbb4507d344426eb398e5ee4891374e 2023-03-20 10:59:01,234 DEBUG [Thread-3746] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:59:01,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:59:01,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:59:01,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:59:01,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:59:01,254 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:59:01,254 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:59:01,254 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:59:01,254 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:59:01,255 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:01,255 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:01,255 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:59:01,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:59:01,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:59:01,255 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:01,256 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,256 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:59:01,257 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:01,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:59:01,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:01,258 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:59:01,267 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:59:01,267 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:59:01,268 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:59:01,268 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:59:01,268 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:59:01,268 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:59:01,268 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:59:01,268 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:59:01,268 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. started... 2023-03-20 10:59:01,268 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. as already flushing 2023-03-20 10:59:01,268 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:59:01,268 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:59:01,268 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:59:01,268 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,268 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:59:01,268 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:59:01,269 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:59:01,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:59:01,270 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:01,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:59:01,270 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:01,270 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:59:01,270 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/A/3062807056ac4d6db02049608bda4853 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/3062807056ac4d6db02049608bda4853 2023-03-20 10:59:01,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:59:01,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:01,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:59:01,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:59:01,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f55c561[Count = 0] remaining members to acquire global barrier 2023-03-20 10:59:01,272 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:59:01,272 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:01,273 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:01,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:01,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:01,273 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:59:01,274 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:59:01,274 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:59:01,274 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,274 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:59:01,276 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,276 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 96c44321a6ff3c73883afe25fc999ea5/A of 96c44321a6ff3c73883afe25fc999ea5 into 3062807056ac4d6db02049608bda4853(size=25.8 K), total size for store is 51.2 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-03-20 10:59:01,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:59:01,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:59:01,276 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:59:01,276 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:59:01,277 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:59:01,276 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 96c44321a6ff3c73883afe25fc999ea5: 2023-03-20 10:59:01,277 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5., storeName=96c44321a6ff3c73883afe25fc999ea5/A, priority=10, startTime=1679309939919; duration=1sec 2023-03-20 10:59:01,277 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:59:01,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:59:01,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:59:01,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:01,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:59:01,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:01,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:59:01,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:59:01,282 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:59:01,282 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:59:01,282 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:59:01,282 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:59:01,282 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:59:01,286 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:01,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:01,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:01,287 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:01,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:01,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:59:01,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:59:01,290 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:59:01,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:59:01,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:59:01,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:59:01,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:01,294 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:59:01,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:01,306 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:01,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:59:01,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:01,317 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:59:01,317 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:59:01,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:59:01,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:59:01,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:59:01,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:59:01,318 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:59:01,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:59:01,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:59:01,318 DEBUG [Thread-3746] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:59:01,318 DEBUG [Thread-3746] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:59:01,319 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:59:01,322 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,322 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:01,322 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:01,322 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,322 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:01,322 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:01,322 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:01,357 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:01,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39904 deadline: 1679310001354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,358 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:01,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39798 deadline: 1679310001357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,418 DEBUG [Thread-3746] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:59:01,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:59:01,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:59:01,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:59:01,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:59:01,439 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:59:01,439 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:59:01,439 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:59:01,439 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:59:01,440 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:01,440 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:01,441 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,441 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,441 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:59:01,442 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:59:01,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:59:01,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:59:01,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:01,443 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:01,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:59:01,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:01,443 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:59:01,450 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:59:01,450 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:59:01,450 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:59:01,450 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:59:01,450 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:59:01,450 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:59:01,450 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:59:01,450 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:59:01,450 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. started... 2023-03-20 10:59:01,451 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. as already flushing 2023-03-20 10:59:01,451 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:59:01,451 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:59:01,451 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:59:01,451 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,451 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:59:01,451 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:59:01,462 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:01,466 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:59:01,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:59:01,467 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:01,467 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:59:01,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:59:01,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:59:01,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:01,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:59:01,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:59:01,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35643049[Count = 0] remaining members to acquire global barrier 2023-03-20 10:59:01,476 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:59:01,476 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:01,478 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:01,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:01,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:01,478 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:59:01,479 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:59:01,479 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:59:01,479 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,479 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:59:01,479 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:59:01,479 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,479 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:59:01,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:59:01,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:59:01,479 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:59:01,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:59:01,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:59:01,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:01,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:59:01,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:01,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:59:01,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:59:01,481 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:59:01,482 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:59:01,482 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:59:01,482 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:59:01,482 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:59:01,483 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:01,483 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:59:01,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:01,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:01,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:59:01,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:59:01,483 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:01,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:01,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:59:01,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:59:01,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:01,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:59:01,484 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:01,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:59:01,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:01,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,485 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:59:01,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:01,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,487 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:59:01,487 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:59:01,487 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:59:01,487 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,487 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:01,487 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:01,487 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,487 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:01,487 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:01,487 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:01,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:59:01,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:59:01,488 DEBUG [Thread-3746] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:59:01,488 DEBUG [Thread-3746] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:59:01,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:59:01,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:59:01,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:59:01,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:59:01,588 DEBUG [Thread-3746] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:59:01,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:59:01,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:59:01,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:59:01,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:59:01,593 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:59:01,593 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:59:01,593 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:59:01,593 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:59:01,593 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:01,593 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:01,597 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:59:01,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:59:01,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:59:01,597 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:01,598 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,598 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:59:01,598 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:01,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:59:01,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:01,598 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:59:01,600 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:59:01,600 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:59:01,600 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:59:01,600 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:59:01,600 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:59:01,601 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:59:01,601 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:59:01,601 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:59:01,601 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. started... 2023-03-20 10:59:01,601 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. as already flushing 2023-03-20 10:59:01,601 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:59:01,601 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:59:01,601 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:59:01,601 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,601 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:59:01,601 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:59:01,602 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:59:01,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:59:01,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:59:01,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:59:01,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:01,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:59:01,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:59:01,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b91ea91[Count = 0] remaining members to acquire global barrier 2023-03-20 10:59:01,604 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:59:01,605 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:01,605 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,605 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:59:01,606 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:01,606 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:01,606 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:01,606 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:59:01,606 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:59:01,606 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:59:01,606 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:59:01,609 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,609 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:59:01,609 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:59:01,609 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:59:01,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:59:01,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:59:01,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:59:01,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:59:01,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:01,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:59:01,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:01,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:59:01,612 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:59:01,612 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:59:01,612 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:59:01,612 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:59:01,612 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:59:01,612 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:59:01,625 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=293 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/B/9613b23480154d6e8bc8a8e0adf5fed1 2023-03-20 10:59:01,627 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:01,627 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:01,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:01,627 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:59:01,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:01,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:59:01,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:59:01,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:01,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:59:01,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:59:01,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:01,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:59:01,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:01,639 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:59:01,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:01,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:59:01,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:01,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,641 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,642 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:59:01,643 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:01,643 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:59:01,643 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:59:01,643 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:59:01,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:59:01,643 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:01,643 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:01,643 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,643 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:01,643 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:01,643 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:01,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:59:01,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:59:01,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:59:01,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:59:01,643 DEBUG [Thread-3746] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:59:01,643 DEBUG [Thread-3746] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:59:01,656 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=293 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/C/6e9f1474590f4a68a3e932991ed4d71b 2023-03-20 10:59:01,661 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/A/9cbb4507d344426eb398e5ee4891374e as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/9cbb4507d344426eb398e5ee4891374e 2023-03-20 10:59:01,663 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/9cbb4507d344426eb398e5ee4891374e, entries=200, sequenceid=293, filesize=32.3 K 2023-03-20 10:59:01,664 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/B/9613b23480154d6e8bc8a8e0adf5fed1 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/9613b23480154d6e8bc8a8e0adf5fed1 2023-03-20 10:59:01,667 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/9613b23480154d6e8bc8a8e0adf5fed1, entries=150, sequenceid=293, filesize=11.9 K 2023-03-20 10:59:01,667 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/C/6e9f1474590f4a68a3e932991ed4d71b as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/6e9f1474590f4a68a3e932991ed4d71b 2023-03-20 10:59:01,676 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/6e9f1474590f4a68a3e932991ed4d71b, entries=150, sequenceid=293, filesize=11.9 K 2023-03-20 10:59:01,677 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=60.38 KB/61830 for 96c44321a6ff3c73883afe25fc999ea5 in 977ms, sequenceid=293, compaction requested=true 2023-03-20 10:59:01,677 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 96c44321a6ff3c73883afe25fc999ea5: 2023-03-20 10:59:01,677 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:59:01,677 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:59:01,677 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 10:59:01,677 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 10:59:01,677 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:59:01,678 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36952 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 10:59:01,678 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): 96c44321a6ff3c73883afe25fc999ea5/B is initiating minor compaction (all files) 2023-03-20 10:59:01,678 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85538 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 10:59:01,678 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 96c44321a6ff3c73883afe25fc999ea5/B in TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:59:01,678 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1912): 96c44321a6ff3c73883afe25fc999ea5/A is initiating minor compaction (all files) 2023-03-20 10:59:01,678 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/653e7e82f9e148b3b0c275e5de0d5427, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/18918dd42766411f90a8b8fd50500f3c, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/9613b23480154d6e8bc8a8e0adf5fed1] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp, totalSize=36.1 K 2023-03-20 10:59:01,678 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 96c44321a6ff3c73883afe25fc999ea5/A in TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:59:01,678 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/3062807056ac4d6db02049608bda4853, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/fda06680636e4a7680027a86919db163, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/9cbb4507d344426eb398e5ee4891374e] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp, totalSize=83.5 K 2023-03-20 10:59:01,678 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 653e7e82f9e148b3b0c275e5de0d5427, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=259, earliestPutTs=1679309939055 2023-03-20 10:59:01,678 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 3062807056ac4d6db02049608bda4853, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=259, earliestPutTs=1679309939055 2023-03-20 10:59:01,679 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 18918dd42766411f90a8b8fd50500f3c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1679309939394 2023-03-20 10:59:01,679 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 9613b23480154d6e8bc8a8e0adf5fed1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=293, earliestPutTs=1679309940068 2023-03-20 10:59:01,679 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting fda06680636e4a7680027a86919db163, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1679309939394 2023-03-20 10:59:01,679 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 9cbb4507d344426eb398e5ee4891374e, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=293, earliestPutTs=1679309940068 2023-03-20 10:59:01,687 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): 96c44321a6ff3c73883afe25fc999ea5#B#compaction#548 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:59:01,701 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] throttle.PressureAwareThroughputController(145): 96c44321a6ff3c73883afe25fc999ea5#A#compaction#549 average throughput is 1.52 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:59:01,743 DEBUG [Thread-3746] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:59:01,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:59:01,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:59:01,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:59:01,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:59:01,747 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:59:01,747 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:59:01,747 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:59:01,747 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:59:01,748 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:01,748 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:01,749 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:59:01,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:59:01,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:59:01,749 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:01,749 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,749 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:59:01,750 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:01,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:59:01,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:01,750 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:59:01,750 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:59:01,750 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:59:01,750 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:59:01,750 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:59:01,750 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:59:01,750 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:59:01,750 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:59:01,750 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:59:01,751 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. started... 2023-03-20 10:59:01,751 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 96c44321a6ff3c73883afe25fc999ea5 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-20 10:59:01,751 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 96c44321a6ff3c73883afe25fc999ea5, store=A 2023-03-20 10:59:01,751 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:59:01,751 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 96c44321a6ff3c73883afe25fc999ea5, store=B 2023-03-20 10:59:01,751 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:59:01,751 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 96c44321a6ff3c73883afe25fc999ea5, store=C 2023-03-20 10:59:01,751 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:59:01,772 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230320cecb3b6108844f86a77d3bd285ae34c1 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320cecb3b6108844f86a77d3bd285ae34c1 2023-03-20 10:59:01,777 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=307, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/A/719dd91fcc1b4e69a3cbf6cae3a9b181 2023-03-20 10:59:01,825 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=307 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/B/bab1ea6e6840444fbd02d25522d7a7b6 2023-03-20 10:59:01,847 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=307 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/C/e75db226afe346edb9e1595284687883 2023-03-20 10:59:01,853 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/A/719dd91fcc1b4e69a3cbf6cae3a9b181 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/719dd91fcc1b4e69a3cbf6cae3a9b181 2023-03-20 10:59:01,859 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/719dd91fcc1b4e69a3cbf6cae3a9b181, entries=150, sequenceid=307, filesize=25.4 K 2023-03-20 10:59:01,860 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/B/bab1ea6e6840444fbd02d25522d7a7b6 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/bab1ea6e6840444fbd02d25522d7a7b6 2023-03-20 10:59:01,864 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/bab1ea6e6840444fbd02d25522d7a7b6, entries=150, sequenceid=307, filesize=11.9 K 2023-03-20 10:59:01,865 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/C/e75db226afe346edb9e1595284687883 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/e75db226afe346edb9e1595284687883 2023-03-20 10:59:01,904 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/e75db226afe346edb9e1595284687883, entries=150, sequenceid=307, filesize=11.9 K 2023-03-20 10:59:01,905 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=0 B/0 for 96c44321a6ff3c73883afe25fc999ea5 in 154ms, sequenceid=307, compaction requested=true 2023-03-20 10:59:01,905 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 96c44321a6ff3c73883afe25fc999ea5: 2023-03-20 10:59:01,905 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:59:01,905 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:59:01,905 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:59:01,905 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,905 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:59:01,905 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:59:01,911 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:01,911 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:01,911 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:59:01,913 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:59:01,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:59:01,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:59:01,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:59:01,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:01,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:59:01,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:59:01,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56a811e7[Count = 0] remaining members to acquire global barrier 2023-03-20 10:59:01,942 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:59:01,942 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:01,946 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:01,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:01,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:01,947 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:59:01,947 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:59:01,947 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:59:01,948 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:59:01,948 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:59:01,948 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:59:01,951 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,958 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:59:01,958 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:59:01,958 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:59:01,958 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:59:01,958 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:59:01,958 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:59:01,958 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:59:01,961 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:01,961 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:01,961 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:59:01,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:01,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:59:01,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:59:01,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:01,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:01,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:59:01,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:59:01,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:59:01,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:01,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:01,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:59:01,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:01,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:59:01,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:01,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,966 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,969 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,970 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,970 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:01,970 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:01,970 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:01,970 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:01,970 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:01,970 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:01,971 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:59:01,971 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:59:01,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:59:01,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:59:01,973 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:59:01,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:59:01,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:59:01,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:59:01,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:59:01,974 DEBUG [Thread-3746] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:59:01,974 DEBUG [Thread-3746] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:59:01,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 96c44321a6ff3c73883afe25fc999ea5 2023-03-20 10:59:01,990 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 96c44321a6ff3c73883afe25fc999ea5 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-20 10:59:01,991 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 96c44321a6ff3c73883afe25fc999ea5, store=A 2023-03-20 10:59:01,991 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:59:01,991 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 96c44321a6ff3c73883afe25fc999ea5, store=B 2023-03-20 10:59:01,991 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:59:01,991 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 96c44321a6ff3c73883afe25fc999ea5, store=C 2023-03-20 10:59:01,991 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:59:02,018 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303205df02a23a2ee4e49a611493be1b7a4ac to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303205df02a23a2ee4e49a611493be1b7a4ac 2023-03-20 10:59:02,032 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=318, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/A/58ad4501bfb54a8d8935bb8be9fc556a 2023-03-20 10:59:02,064 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679309942048, max timestamp=1679309942062Num uniques 859; 2023-03-20 10:59:02,064 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-20 10:59:02,074 DEBUG [Thread-3746] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:59:02,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:59:02,074 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:02,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39904 deadline: 1679310002072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:02,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:59:02,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:59:02,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:59:02,078 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:59:02,078 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:59:02,078 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:59:02,078 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:59:02,078 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:02,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39798 deadline: 1679310002075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:02,079 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:02,079 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:02,079 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:59:02,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:59:02,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:59:02,079 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:02,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:02,080 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:02,080 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:02,080 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:59:02,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:59:02,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:02,080 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:59:02,082 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:59:02,082 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:59:02,082 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:59:02,082 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:59:02,082 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:59:02,086 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:59:02,086 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:59:02,086 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:59:02,086 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. started... 2023-03-20 10:59:02,086 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. as already flushing 2023-03-20 10:59:02,087 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:59:02,087 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:59:02,087 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:59:02,087 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:02,087 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:59:02,087 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:59:02,088 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:02,088 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:02,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:02,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:59:02,088 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:02,088 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:59:02,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:59:02,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:59:02,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:59:02,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:02,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:02,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:59:02,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:59:02,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1797b89e[Count = 0] remaining members to acquire global barrier 2023-03-20 10:59:02,090 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:59:02,090 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:02,091 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:02,091 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:02,091 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:59:02,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:02,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:02,092 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:59:02,092 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:59:02,092 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:59:02,100 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:59:02,100 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:59:02,100 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:59:02,100 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:02,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:02,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:59:02,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:59:02,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:59:02,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:59:02,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:02,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:02,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:59:02,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:02,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:02,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:59:02,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:59:02,105 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:59:02,105 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:59:02,105 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:59:02,105 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:59:02,105 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:59:02,106 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/B/b007d2fddf404bba91c71abb80450208 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/b007d2fddf404bba91c71abb80450208 2023-03-20 10:59:02,111 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:02,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:02,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:59:02,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:59:02,111 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:02,111 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:59:02,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:02,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:02,111 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:02,114 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:02,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:59:02,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:59:02,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:59:02,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:02,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:02,116 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 96c44321a6ff3c73883afe25fc999ea5/B of 96c44321a6ff3c73883afe25fc999ea5 into b007d2fddf404bba91c71abb80450208(size=12.4 K), total size for store is 24.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:59:02,116 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 96c44321a6ff3c73883afe25fc999ea5: 2023-03-20 10:59:02,117 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5., storeName=96c44321a6ff3c73883afe25fc999ea5/B, priority=13, startTime=1679309941677; duration=0sec 2023-03-20 10:59:02,117 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:59:02,117 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-20 10:59:02,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:59:02,120 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:59:02,120 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:59:02,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:59:02,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:59:02,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:59:02,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:59:02,121 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:02,121 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:02,121 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:02,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:59:02,121 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:02,121 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:02,121 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:02,121 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:02,121 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:59:02,123 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:59:02,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:59:02,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:59:02,123 DEBUG [Thread-3746] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:59:02,123 DEBUG [Thread-3746] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:59:02,125 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49104 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-20 10:59:02,125 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): 96c44321a6ff3c73883afe25fc999ea5/C is initiating minor compaction (all files) 2023-03-20 10:59:02,125 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 96c44321a6ff3c73883afe25fc999ea5/C in TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:59:02,125 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/3fbf0c8c0d324c47a7daa80b2a593e8d, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/a01c3fdcd28c48d9855ba61d4957b4cd, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/6e9f1474590f4a68a3e932991ed4d71b, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/e75db226afe346edb9e1595284687883] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp, totalSize=48.0 K 2023-03-20 10:59:02,127 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 3fbf0c8c0d324c47a7daa80b2a593e8d, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=259, earliestPutTs=1679309939055 2023-03-20 10:59:02,127 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting a01c3fdcd28c48d9855ba61d4957b4cd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1679309939394 2023-03-20 10:59:02,127 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 6e9f1474590f4a68a3e932991ed4d71b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=293, earliestPutTs=1679309940068 2023-03-20 10:59:02,127 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting e75db226afe346edb9e1595284687883, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=307, earliestPutTs=1679309940720 2023-03-20 10:59:02,150 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): 96c44321a6ff3c73883afe25fc999ea5#C#compaction#555 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:59:02,180 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:02,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39904 deadline: 1679310002176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:02,186 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:02,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39798 deadline: 1679310002183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:02,217 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/A/9931de83d4eb48b1afbfe38048254fb0 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/9931de83d4eb48b1afbfe38048254fb0 2023-03-20 10:59:02,223 DEBUG [Thread-3746] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:59:02,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:59:02,226 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 96c44321a6ff3c73883afe25fc999ea5/A of 96c44321a6ff3c73883afe25fc999ea5 into 9931de83d4eb48b1afbfe38048254fb0(size=25.9 K), total size for store is 51.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:59:02,226 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 96c44321a6ff3c73883afe25fc999ea5: 2023-03-20 10:59:02,226 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5., storeName=96c44321a6ff3c73883afe25fc999ea5/A, priority=13, startTime=1679309941677; duration=0sec 2023-03-20 10:59:02,226 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:59:02,227 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:59:02,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:59:02,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:59:02,244 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:59:02,244 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:59:02,244 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:59:02,244 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:59:02,247 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:02,247 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:02,248 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:59:02,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:59:02,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:59:02,248 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:02,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:02,249 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:02,249 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:59:02,249 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:02,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:59:02,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:02,249 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:59:02,250 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:59:02,250 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:59:02,250 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:59:02,250 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:59:02,250 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:59:02,250 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:59:02,250 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:59:02,250 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:59:02,250 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. started... 2023-03-20 10:59:02,250 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. as already flushing 2023-03-20 10:59:02,250 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:59:02,250 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:59:02,250 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:59:02,251 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:02,251 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:59:02,251 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:59:02,252 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:02,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:02,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:59:02,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:59:02,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:59:02,253 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:02,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:59:02,253 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:02,254 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:59:02,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:02,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:02,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:59:02,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:59:02,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e5345d[Count = 0] remaining members to acquire global barrier 2023-03-20 10:59:02,256 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:59:02,256 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:02,264 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:02,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:02,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:02,264 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:59:02,264 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:59:02,264 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:59:02,269 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:59:02,269 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:59:02,269 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:59:02,274 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:02,277 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:59:02,277 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:59:02,277 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:59:02,277 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:59:02,277 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:59:02,277 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:59:02,277 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:59:02,281 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:02,281 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:59:02,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:02,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:02,281 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:02,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:02,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:59:02,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:59:02,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:59:02,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:59:02,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:59:02,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:02,285 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:02,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:02,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:59:02,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:02,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:02,289 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:02,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:59:02,296 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:02,296 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:02,296 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:02,296 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:02,296 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:02,296 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:02,296 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:02,296 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:59:02,296 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:59:02,296 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:59:02,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:59:02,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:59:02,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:59:02,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:59:02,297 DEBUG [Thread-3746] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:59:02,297 DEBUG [Thread-3746] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:59:02,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:59:02,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:59:02,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:02,300 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:59:02,385 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:02,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39904 deadline: 1679310002382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:02,389 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:02,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39798 deadline: 1679310002387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:02,397 DEBUG [Thread-3746] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:59:02,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:59:02,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:59:02,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:59:02,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:59:02,414 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:59:02,414 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:59:02,414 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:59:02,414 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:59:02,418 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:02,418 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:02,426 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:59:02,426 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:02,426 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:02,426 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:59:02,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:59:02,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:59:02,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:02,434 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:02,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:59:02,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:02,435 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:59:02,435 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:59:02,435 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:59:02,435 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:59:02,435 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:59:02,435 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:59:02,436 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:59:02,436 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:59:02,436 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:59:02,436 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. started... 2023-03-20 10:59:02,436 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. as already flushing 2023-03-20 10:59:02,436 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:59:02,436 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:59:02,436 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:59:02,436 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:02,436 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:59:02,436 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:59:02,437 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:02,437 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:02,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:02,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:59:02,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:59:02,437 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:02,437 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:59:02,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:59:02,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:59:02,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:02,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:02,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:59:02,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:59:02,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@577493d2[Count = 0] remaining members to acquire global barrier 2023-03-20 10:59:02,439 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:59:02,439 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:02,441 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:02,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:02,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:02,441 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:02,441 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:59:02,441 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:59:02,441 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:59:02,441 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:59:02,442 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:02,442 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:59:02,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:02,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:59:02,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:59:02,442 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:59:02,442 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:59:02,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:59:02,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:59:02,456 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=318 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/B/4509c2b8da064dc9a6c83c2f3ce4f0dd 2023-03-20 10:59:02,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:02,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:02,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:59:02,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:02,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:02,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:59:02,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:59:02,461 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:59:02,461 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:59:02,461 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:59:02,461 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:59:02,461 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:59:02,464 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:02,464 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:59:02,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:02,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:02,464 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:02,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:02,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:59:02,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:59:02,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:59:02,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:59:02,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:02,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:59:02,467 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:02,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:02,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:59:02,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:02,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:02,469 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:02,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:59:02,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:02,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:02,478 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:59:02,478 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:59:02,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:59:02,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:59:02,478 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:02,478 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:59:02,478 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:02,478 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:59:02,478 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:59:02,478 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:02,479 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:02,479 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:02,479 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:02,479 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:02,479 DEBUG [Thread-3746] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:59:02,479 DEBUG [Thread-3746] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:59:02,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:59:02,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:59:02,521 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=318 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/C/efb22f1cb50f4667883e7fc7fdbae341 2023-03-20 10:59:02,546 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/A/58ad4501bfb54a8d8935bb8be9fc556a as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/58ad4501bfb54a8d8935bb8be9fc556a 2023-03-20 10:59:02,558 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/58ad4501bfb54a8d8935bb8be9fc556a, entries=400, sequenceid=318, filesize=59.8 K 2023-03-20 10:59:02,559 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/B/4509c2b8da064dc9a6c83c2f3ce4f0dd as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/4509c2b8da064dc9a6c83c2f3ce4f0dd 2023-03-20 10:59:02,568 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/4509c2b8da064dc9a6c83c2f3ce4f0dd, entries=150, sequenceid=318, filesize=11.9 K 2023-03-20 10:59:02,569 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/C/efb22f1cb50f4667883e7fc7fdbae341 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/efb22f1cb50f4667883e7fc7fdbae341 2023-03-20 10:59:02,579 DEBUG [Thread-3746] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:59:02,585 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/efb22f1cb50f4667883e7fc7fdbae341, entries=150, sequenceid=318, filesize=11.9 K 2023-03-20 10:59:02,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:59:02,586 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=125.23 KB/128240 for 96c44321a6ff3c73883afe25fc999ea5 in 596ms, sequenceid=318, compaction requested=true 2023-03-20 10:59:02,586 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 96c44321a6ff3c73883afe25fc999ea5: 2023-03-20 10:59:02,587 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 10:59:02,587 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:59:02,587 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:59:02,587 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-20 10:59:02,588 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 113858 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 10:59:02,588 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1912): 96c44321a6ff3c73883afe25fc999ea5/A is initiating minor compaction (all files) 2023-03-20 10:59:02,588 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 96c44321a6ff3c73883afe25fc999ea5/A in TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:59:02,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:59:02,588 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/9931de83d4eb48b1afbfe38048254fb0, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/719dd91fcc1b4e69a3cbf6cae3a9b181, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/58ad4501bfb54a8d8935bb8be9fc556a] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp, totalSize=111.2 K 2023-03-20 10:59:02,589 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 9931de83d4eb48b1afbfe38048254fb0, keycount=150, bloomtype=ROW, size=25.9 K, encoding=NONE, compression=NONE, seqNum=293, earliestPutTs=1679309940068 2023-03-20 10:59:02,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:59:02,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:59:02,590 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:59:02,590 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:59:02,590 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:59:02,590 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:59:02,590 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 719dd91fcc1b4e69a3cbf6cae3a9b181, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=307, earliestPutTs=1679309940720 2023-03-20 10:59:02,590 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 58ad4501bfb54a8d8935bb8be9fc556a, keycount=400, bloomtype=ROW, size=59.8 K, encoding=NONE, compression=NONE, seqNum=318, earliestPutTs=1679309941933 2023-03-20 10:59:02,591 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:02,591 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:02,592 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:59:02,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:59:02,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:59:02,592 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:02,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:02,593 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:02,593 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:59:02,594 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:02,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:59:02,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:02,594 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:59:02,595 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:59:02,595 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:59:02,595 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:59:02,595 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:59:02,595 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:59:02,595 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:59:02,595 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:59:02,597 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:59:02,597 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. started... 2023-03-20 10:59:02,598 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 96c44321a6ff3c73883afe25fc999ea5 3/3 column families, dataSize=125.23 KB heapSize=366.80 KB 2023-03-20 10:59:02,600 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 96c44321a6ff3c73883afe25fc999ea5, store=A 2023-03-20 10:59:02,600 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:59:02,600 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 96c44321a6ff3c73883afe25fc999ea5, store=B 2023-03-20 10:59:02,600 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:59:02,603 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 96c44321a6ff3c73883afe25fc999ea5, store=C 2023-03-20 10:59:02,603 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:59:02,621 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/C/bd68d316ebc94422b631eb0c304a524d as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/bd68d316ebc94422b631eb0c304a524d 2023-03-20 10:59:02,632 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 96c44321a6ff3c73883afe25fc999ea5/C of 96c44321a6ff3c73883afe25fc999ea5 into bd68d316ebc94422b631eb0c304a524d(size=12.4 K), total size for store is 24.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:59:02,632 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 96c44321a6ff3c73883afe25fc999ea5: 2023-03-20 10:59:02,632 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5., storeName=96c44321a6ff3c73883afe25fc999ea5/C, priority=12, startTime=1679309941677; duration=0sec 2023-03-20 10:59:02,632 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-20 10:59:02,632 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-20 10:59:02,635 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37104 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-20 10:59:02,635 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): 96c44321a6ff3c73883afe25fc999ea5/B is initiating minor compaction (all files) 2023-03-20 10:59:02,635 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 96c44321a6ff3c73883afe25fc999ea5/B in TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:59:02,635 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/b007d2fddf404bba91c71abb80450208, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/bab1ea6e6840444fbd02d25522d7a7b6, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/4509c2b8da064dc9a6c83c2f3ce4f0dd] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp, totalSize=36.2 K 2023-03-20 10:59:02,635 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting b007d2fddf404bba91c71abb80450208, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=293, earliestPutTs=1679309940068 2023-03-20 10:59:02,635 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting bab1ea6e6840444fbd02d25522d7a7b6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=307, earliestPutTs=1679309940720 2023-03-20 10:59:02,636 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 4509c2b8da064dc9a6c83c2f3ce4f0dd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=318, earliestPutTs=1679309941984 2023-03-20 10:59:02,667 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): 96c44321a6ff3c73883afe25fc999ea5#B#compaction#558 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:59:02,688 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] throttle.PressureAwareThroughputController(145): 96c44321a6ff3c73883afe25fc999ea5#A#compaction#557 average throughput is 0.30 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:59:02,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 96c44321a6ff3c73883afe25fc999ea5 2023-03-20 10:59:02,694 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. as already flushing 2023-03-20 10:59:02,717 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:02,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39904 deadline: 1679310002714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:02,720 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:02,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39798 deadline: 1679310002718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:02,767 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/B/b2a986bfe74a48239c2cf212a40504d7 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/b2a986bfe74a48239c2cf212a40504d7 2023-03-20 10:59:02,779 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303201085b13b9ce34cc59289601f30ded8a9 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303201085b13b9ce34cc59289601f30ded8a9 2023-03-20 10:59:02,790 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=346, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/A/367d20b9632a448291d5e42fb2586fea 2023-03-20 10:59:02,790 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 96c44321a6ff3c73883afe25fc999ea5/B of 96c44321a6ff3c73883afe25fc999ea5 into b2a986bfe74a48239c2cf212a40504d7(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:59:02,790 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 96c44321a6ff3c73883afe25fc999ea5: 2023-03-20 10:59:02,790 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5., storeName=96c44321a6ff3c73883afe25fc999ea5/B, priority=13, startTime=1679309942587; duration=0sec 2023-03-20 10:59:02,790 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:59:02,790 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-20 10:59:02,798 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-20 10:59:02,798 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-20 10:59:02,798 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. because compaction request was cancelled 2023-03-20 10:59:02,822 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:02,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39904 deadline: 1679310002819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:02,823 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:02,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39798 deadline: 1679310002821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:02,883 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=346 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/B/55a3321f2c514e2588cfe8d12ffc951a 2023-03-20 10:59:03,029 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:03,029 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:03,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39798 deadline: 1679310003025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:03,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39904 deadline: 1679310003025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:03,335 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:03,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39798 deadline: 1679310003332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:03,335 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:03,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39904 deadline: 1679310003334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:03,348 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=346 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/C/c4f70332d4624094a04d4d39c2489c4b 2023-03-20 10:59:03,358 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/A/367d20b9632a448291d5e42fb2586fea as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/367d20b9632a448291d5e42fb2586fea 2023-03-20 10:59:03,362 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/367d20b9632a448291d5e42fb2586fea, entries=150, sequenceid=346, filesize=25.4 K 2023-03-20 10:59:03,363 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/B/55a3321f2c514e2588cfe8d12ffc951a as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/55a3321f2c514e2588cfe8d12ffc951a 2023-03-20 10:59:03,369 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/55a3321f2c514e2588cfe8d12ffc951a, entries=150, sequenceid=346, filesize=11.9 K 2023-03-20 10:59:03,370 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/C/c4f70332d4624094a04d4d39c2489c4b as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/c4f70332d4624094a04d4d39c2489c4b 2023-03-20 10:59:03,378 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/c4f70332d4624094a04d4d39c2489c4b, entries=150, sequenceid=346, filesize=11.9 K 2023-03-20 10:59:03,379 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~125.23 KB/128240, heapSize ~366.75 KB/375552, currentSize=53.67 KB/54960 for 96c44321a6ff3c73883afe25fc999ea5 in 781ms, sequenceid=346, compaction requested=true 2023-03-20 10:59:03,379 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 96c44321a6ff3c73883afe25fc999ea5: 2023-03-20 10:59:03,379 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:59:03,379 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:59:03,379 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:59:03,379 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:03,379 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:59:03,379 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:59:03,380 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:03,381 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:03,381 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:59:03,381 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:03,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:03,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:59:03,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:59:03,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:59:03,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:59:03,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:03,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:03,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:59:03,382 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:59:03,382 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41c597f2[Count = 0] remaining members to acquire global barrier 2023-03-20 10:59:03,382 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:59:03,382 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:03,383 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:03,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:03,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:03,383 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:59:03,383 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:59:03,383 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:59:03,384 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:03,384 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:59:03,384 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:59:03,384 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:59:03,384 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:59:03,384 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:03,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:03,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:59:03,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:59:03,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:59:03,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:59:03,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:03,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:03,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:59:03,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:03,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:03,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:59:03,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:59:03,386 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:59:03,386 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:59:03,386 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:59:03,386 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:59:03,386 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:59:03,387 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:03,387 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:03,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:03,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:59:03,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:59:03,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:03,387 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:59:03,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:03,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:59:03,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:59:03,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:59:03,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:03,388 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:03,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:03,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:59:03,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:03,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:03,389 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:03,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:59:03,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:03,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:03,392 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:59:03,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:59:03,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:59:03,392 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:59:03,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:59:03,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:59:03,393 DEBUG [Thread-3746] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:59:03,393 DEBUG [Thread-3746] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:59:03,392 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:59:03,392 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:59:03,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:59:03,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:59:03,393 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:03,393 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:03,393 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:03,393 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:03,393 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:03,393 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:03,393 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:03,493 DEBUG [Thread-3746] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:59:03,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:59:03,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:59:03,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:59:03,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:59:03,495 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:59:03,495 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:59:03,495 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:59:03,496 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:59:03,496 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:03,496 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:03,497 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:59:03,497 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:03,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:59:03,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:59:03,497 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:03,497 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:59:03,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:03,498 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:03,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:59:03,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:03,498 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:59:03,499 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:59:03,499 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:59:03,499 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:59:03,499 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:59:03,499 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:59:03,499 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:59:03,499 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:59:03,499 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. started... 2023-03-20 10:59:03,499 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 96c44321a6ff3c73883afe25fc999ea5 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-20 10:59:03,499 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 96c44321a6ff3c73883afe25fc999ea5, store=A 2023-03-20 10:59:03,499 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:59:03,499 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 96c44321a6ff3c73883afe25fc999ea5, store=B 2023-03-20 10:59:03,500 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:59:03,500 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 96c44321a6ff3c73883afe25fc999ea5, store=C 2023-03-20 10:59:03,500 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:59:03,560 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/A/eb3a60c80a45422a9ddd115c8b827b71 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/eb3a60c80a45422a9ddd115c8b827b71 2023-03-20 10:59:03,566 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 96c44321a6ff3c73883afe25fc999ea5/A of 96c44321a6ff3c73883afe25fc999ea5 into eb3a60c80a45422a9ddd115c8b827b71(size=26.0 K), total size for store is 51.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:59:03,566 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 96c44321a6ff3c73883afe25fc999ea5: 2023-03-20 10:59:03,566 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5., storeName=96c44321a6ff3c73883afe25fc999ea5/A, priority=13, startTime=1679309942587; duration=0sec 2023-03-20 10:59:03,566 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:59:03,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 96c44321a6ff3c73883afe25fc999ea5 2023-03-20 10:59:03,843 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. as already flushing 2023-03-20 10:59:03,911 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679309943893, max timestamp=1679309943905Num uniques 859; 2023-03-20 10:59:03,911 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-20 10:59:03,913 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679309943893, max timestamp=1679309943905Num uniques 859; 2023-03-20 10:59:03,913 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-20 10:59:03,914 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679309943893, max timestamp=1679309943905Num uniques 859; 2023-03-20 10:59:03,914 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase20:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-20 10:59:03,939 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303201ba4512296df40d68294e8cb4b2ba8be to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303201ba4512296df40d68294e8cb4b2ba8be 2023-03-20 10:59:03,963 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:03,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39904 deadline: 1679310003958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:03,967 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:03,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39798 deadline: 1679310003963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:04,067 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:04,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39904 deadline: 1679310004064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:04,070 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:04,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39798 deadline: 1679310004069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:04,128 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:04,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39896 deadline: 1679310004126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:04,129 DEBUG [Thread-3742] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8183 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5., hostname=jenkins-hbase20.apache.org,36041,1679309777062, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-20 10:59:04,270 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:04,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39904 deadline: 1679310004268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:04,274 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:04,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39798 deadline: 1679310004271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:04,346 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=359, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/A/2372c47596314bb3aea73b00615d948e 2023-03-20 10:59:04,575 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:04,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39904 deadline: 1679310004573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:04,578 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:04,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39798 deadline: 1679310004576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:04,753 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=359 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/B/a5892e075c35473c8fd9ef61c2fa0bfd 2023-03-20 10:59:04,781 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=359 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/C/3134d0ff6c354e81b0627dd51f227a86 2023-03-20 10:59:04,786 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/A/2372c47596314bb3aea73b00615d948e as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/2372c47596314bb3aea73b00615d948e 2023-03-20 10:59:04,790 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/2372c47596314bb3aea73b00615d948e, entries=100, sequenceid=359, filesize=18.6 K 2023-03-20 10:59:04,790 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/B/a5892e075c35473c8fd9ef61c2fa0bfd as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/a5892e075c35473c8fd9ef61c2fa0bfd 2023-03-20 10:59:04,793 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/a5892e075c35473c8fd9ef61c2fa0bfd, entries=100, sequenceid=359, filesize=9.5 K 2023-03-20 10:59:04,794 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/C/3134d0ff6c354e81b0627dd51f227a86 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/3134d0ff6c354e81b0627dd51f227a86 2023-03-20 10:59:04,797 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/3134d0ff6c354e81b0627dd51f227a86, entries=100, sequenceid=359, filesize=9.5 K 2023-03-20 10:59:04,798 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 96c44321a6ff3c73883afe25fc999ea5 in 1299ms, sequenceid=359, compaction requested=true 2023-03-20 10:59:04,798 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 96c44321a6ff3c73883afe25fc999ea5: 2023-03-20 10:59:04,798 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:59:04,798 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:59:04,798 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:59:04,798 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:04,802 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:59:04,802 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:59:04,803 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:04,803 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:04,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:04,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:59:04,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:59:04,803 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:04,803 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:59:04,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:59:04,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:59:04,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:04,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:04,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:59:04,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:59:04,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6bbc941c[Count = 0] remaining members to acquire global barrier 2023-03-20 10:59:04,804 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:59:04,805 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:04,805 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:04,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:04,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:04,805 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:59:04,805 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:59:04,805 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:59:04,805 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:04,805 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:59:04,806 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:04,806 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:59:04,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:04,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:59:04,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:59:04,806 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:59:04,806 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:59:04,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:59:04,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:59:04,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:04,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:04,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:59:04,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:04,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:04,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:59:04,808 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:59:04,808 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:59:04,808 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:59:04,808 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:59:04,808 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:59:04,808 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:59:04,809 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:04,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:04,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:59:04,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:59:04,809 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:04,809 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:59:04,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:04,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:04,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:59:04,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:59:04,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:59:04,810 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:04,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:04,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:04,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:59:04,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:04,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:04,810 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:04,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:59:04,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:04,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:04,812 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:04,812 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:04,812 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:04,812 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:04,812 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:04,812 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:04,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:59:04,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:59:04,812 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:59:04,812 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:04,812 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:59:04,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:59:04,812 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:59:04,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:59:04,812 DEBUG [Thread-3746] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:59:04,813 DEBUG [Thread-3746] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:59:04,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:59:04,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:59:04,913 DEBUG [Thread-3746] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:59:04,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:59:04,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:59:04,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:59:04,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:59:04,921 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:59:04,921 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:59:04,921 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:59:04,921 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:59:04,922 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:04,922 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:04,923 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:59:04,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:59:04,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:59:04,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:04,924 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:04,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:59:04,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:04,924 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:59:04,924 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:59:04,924 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:59:04,924 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:59:04,924 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:59:04,924 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:59:04,925 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:59:04,925 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:59:04,925 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:59:04,925 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. started... 2023-03-20 10:59:04,925 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 96c44321a6ff3c73883afe25fc999ea5 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-03-20 10:59:04,925 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:04,925 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 96c44321a6ff3c73883afe25fc999ea5, store=A 2023-03-20 10:59:04,925 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:59:04,925 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 96c44321a6ff3c73883afe25fc999ea5, store=B 2023-03-20 10:59:04,925 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:59:04,925 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 96c44321a6ff3c73883afe25fc999ea5, store=C 2023-03-20 10:59:04,925 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-20 10:59:04,925 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:04,925 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:59:04,983 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303204ba1bcc562b9435ca34124cc47f41df2 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303204ba1bcc562b9435ca34124cc47f41df2 2023-03-20 10:59:05,022 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=396, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/A/3a579176ad084af0ba98771ac9317d94 2023-03-20 10:59:05,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 96c44321a6ff3c73883afe25fc999ea5 2023-03-20 10:59:05,081 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. as already flushing 2023-03-20 10:59:05,139 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:05,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39904 deadline: 1679310005129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:05,140 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:05,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39798 deadline: 1679310005134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:05,247 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:05,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39904 deadline: 1679310005245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:05,248 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:05,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39798 deadline: 1679310005245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:05,450 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:05,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39904 deadline: 1679310005449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:05,455 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:05,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39798 deadline: 1679310005451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:05,479 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=396 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/B/38da7807ab7e460aa877f2d14dc18168 2023-03-20 10:59:05,756 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:05,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39904 deadline: 1679310005752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:05,762 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:05,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39798 deadline: 1679310005759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:05,936 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=396 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/C/d38f0fee530c4b7ebecbd4a75dd06446 2023-03-20 10:59:05,940 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/A/3a579176ad084af0ba98771ac9317d94 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/3a579176ad084af0ba98771ac9317d94 2023-03-20 10:59:05,947 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/3a579176ad084af0ba98771ac9317d94, entries=150, sequenceid=396, filesize=25.4 K 2023-03-20 10:59:05,948 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/B/38da7807ab7e460aa877f2d14dc18168 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/38da7807ab7e460aa877f2d14dc18168 2023-03-20 10:59:05,966 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/38da7807ab7e460aa877f2d14dc18168, entries=150, sequenceid=396, filesize=11.9 K 2023-03-20 10:59:05,967 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/C/d38f0fee530c4b7ebecbd4a75dd06446 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/d38f0fee530c4b7ebecbd4a75dd06446 2023-03-20 10:59:05,976 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/d38f0fee530c4b7ebecbd4a75dd06446, entries=150, sequenceid=396, filesize=11.9 K 2023-03-20 10:59:05,976 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=60.38 KB/61830 for 96c44321a6ff3c73883afe25fc999ea5 in 1051ms, sequenceid=396, compaction requested=true 2023-03-20 10:59:05,976 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 96c44321a6ff3c73883afe25fc999ea5: 2023-03-20 10:59:05,976 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:59:05,976 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:59:05,977 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:59:05,977 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:05,977 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:59:05,977 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:59:05,980 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:05,980 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:05,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:05,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:59:05,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:59:05,981 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:05,981 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:59:05,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:59:05,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:59:05,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:05,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:05,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:59:05,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:59:05,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41d112ea[Count = 0] remaining members to acquire global barrier 2023-03-20 10:59:05,984 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:59:05,984 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:05,994 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:05,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:05,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:05,994 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:59:05,994 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:59:05,994 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:59:05,995 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:05,995 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:59:05,996 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:59:05,996 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:59:05,996 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:59:05,996 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:05,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:05,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:59:05,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:59:05,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:59:05,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:59:05,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:05,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:05,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:59:05,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:05,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:06,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:59:06,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:59:06,002 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:59:06,002 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:59:06,002 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:59:06,003 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:59:06,003 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:59:06,011 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:06,011 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:06,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:06,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:06,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:59:06,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:59:06,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:06,011 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:59:06,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:59:06,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:59:06,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:06,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:59:06,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:06,015 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:06,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:59:06,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:06,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:06,020 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:06,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:59:06,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:06,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:06,022 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:06,022 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:06,022 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:06,022 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:06,022 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:06,023 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:59:06,023 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:59:06,023 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:06,023 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:59:06,023 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:06,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:59:06,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:59:06,023 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:59:06,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:59:06,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:59:06,023 DEBUG [Thread-3746] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:59:06,023 DEBUG [Thread-3746] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:59:06,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:59:06,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:59:06,123 DEBUG [Thread-3746] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:59:06,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:59:06,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:59:06,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:59:06,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:59:06,139 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:59:06,139 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:59:06,139 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:59:06,139 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:59:06,139 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:06,139 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:06,140 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:59:06,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:59:06,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:59:06,140 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:06,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:06,140 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:06,140 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:59:06,141 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:06,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:59:06,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:06,141 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:59:06,141 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:59:06,141 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:59:06,141 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:59:06,141 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:59:06,141 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:59:06,150 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:59:06,150 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:59:06,150 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:59:06,150 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. started... 2023-03-20 10:59:06,150 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 96c44321a6ff3c73883afe25fc999ea5 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-20 10:59:06,150 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 96c44321a6ff3c73883afe25fc999ea5, store=A 2023-03-20 10:59:06,150 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:59:06,150 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 96c44321a6ff3c73883afe25fc999ea5, store=B 2023-03-20 10:59:06,150 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:59:06,151 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 96c44321a6ff3c73883afe25fc999ea5, store=C 2023-03-20 10:59:06,151 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:59:06,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 96c44321a6ff3c73883afe25fc999ea5 2023-03-20 10:59:06,265 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. as already flushing 2023-03-20 10:59:06,325 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:06,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39904 deadline: 1679310006322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:06,326 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:06,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39798 deadline: 1679310006324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:06,432 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:06,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39904 deadline: 1679310006428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:06,432 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:06,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39798 deadline: 1679310006428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:06,596 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303204677950d58c34e29be5b187ec5b529bc to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303204677950d58c34e29be5b187ec5b529bc 2023-03-20 10:59:06,639 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:06,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39798 deadline: 1679310006637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:06,643 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:06,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39904 deadline: 1679310006640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:06,944 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:06,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39798 deadline: 1679310006941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:06,950 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:06,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39904 deadline: 1679310006946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:07,019 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=408, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/A/d6e0709fdbbb411e821a84f481fc3ed9 2023-03-20 10:59:07,428 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=408 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/B/3cda0a4a934e47abb159565f9e54c34a 2023-03-20 10:59:07,448 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:07,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39798 deadline: 1679310007445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:07,462 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:07,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39904 deadline: 1679310007457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:07,880 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=408 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/C/46e69a19113e4249abc471d7779d34f3 2023-03-20 10:59:07,887 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/A/d6e0709fdbbb411e821a84f481fc3ed9 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/d6e0709fdbbb411e821a84f481fc3ed9 2023-03-20 10:59:07,892 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/d6e0709fdbbb411e821a84f481fc3ed9, entries=150, sequenceid=408, filesize=25.4 K 2023-03-20 10:59:07,893 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/B/3cda0a4a934e47abb159565f9e54c34a as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/3cda0a4a934e47abb159565f9e54c34a 2023-03-20 10:59:07,906 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/3cda0a4a934e47abb159565f9e54c34a, entries=150, sequenceid=408, filesize=11.9 K 2023-03-20 10:59:07,916 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/C/46e69a19113e4249abc471d7779d34f3 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/46e69a19113e4249abc471d7779d34f3 2023-03-20 10:59:07,927 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/46e69a19113e4249abc471d7779d34f3, entries=150, sequenceid=408, filesize=11.9 K 2023-03-20 10:59:07,928 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 96c44321a6ff3c73883afe25fc999ea5 in 1778ms, sequenceid=408, compaction requested=true 2023-03-20 10:59:07,928 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 96c44321a6ff3c73883afe25fc999ea5: 2023-03-20 10:59:07,928 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:59:07,928 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:59:07,928 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:59:07,928 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:07,928 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:59:07,928 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:59:07,929 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:07,929 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:07,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:07,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:59:07,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:59:07,930 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:07,930 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:59:07,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:59:07,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:59:07,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:07,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:07,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:59:07,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:59:07,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32c8e301[Count = 0] remaining members to acquire global barrier 2023-03-20 10:59:07,931 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:59:07,931 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:07,932 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:07,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:07,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:07,932 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:59:07,932 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:59:07,932 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:59:07,932 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:07,932 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:59:07,933 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:59:07,933 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:59:07,933 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:07,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:07,933 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:59:07,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:59:07,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:59:07,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:59:07,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:59:07,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:07,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:07,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:59:07,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:07,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:07,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:59:07,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:59:07,935 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:59:07,935 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:59:07,935 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:59:07,935 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:59:07,935 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:59:07,936 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:07,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:07,936 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:07,936 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:59:07,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:07,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:07,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:59:07,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:59:07,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:59:07,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:59:07,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:07,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:59:07,937 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:07,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:07,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:59:07,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:07,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:07,938 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:07,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:59:07,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:07,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:07,942 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:07,942 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:07,942 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:07,942 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:07,942 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:59:07,942 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:07,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:59:07,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:59:07,942 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:59:07,942 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:07,942 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:07,943 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:59:07,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:59:07,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:59:07,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:59:07,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:59:07,943 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:59:07,943 DEBUG [Thread-3746] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:59:07,943 DEBUG [Thread-3746] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:59:08,043 DEBUG [Thread-3746] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:59:08,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:59:08,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:59:08,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:59:08,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:59:08,048 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:59:08,048 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:59:08,048 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:59:08,048 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:59:08,049 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:08,049 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:08,050 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:59:08,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:59:08,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:59:08,051 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:08,052 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:08,052 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:59:08,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:08,052 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:08,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:59:08,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:08,052 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:59:08,053 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:59:08,053 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:59:08,053 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:59:08,053 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:59:08,053 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:59:08,053 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:59:08,053 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:59:08,053 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:59:08,053 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. started... 2023-03-20 10:59:08,054 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 96c44321a6ff3c73883afe25fc999ea5 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-20 10:59:08,054 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 96c44321a6ff3c73883afe25fc999ea5, store=A 2023-03-20 10:59:08,054 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:59:08,054 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 96c44321a6ff3c73883afe25fc999ea5, store=B 2023-03-20 10:59:08,054 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:59:08,054 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 96c44321a6ff3c73883afe25fc999ea5, store=C 2023-03-20 10:59:08,054 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:59:08,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 96c44321a6ff3c73883afe25fc999ea5 2023-03-20 10:59:08,457 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. as already flushing 2023-03-20 10:59:08,481 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:08,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39904 deadline: 1679310008478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:08,482 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303208888edf6d56b4b7e8ed63f886a504f96 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303208888edf6d56b4b7e8ed63f886a504f96 2023-03-20 10:59:08,483 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:08,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39798 deadline: 1679310008481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:08,493 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=428, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/A/25593a0c325746188be574597a8934b7 2023-03-20 10:59:08,588 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:08,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39798 deadline: 1679310008587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:08,589 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:08,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39904 deadline: 1679310008586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:08,793 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:08,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39798 deadline: 1679310008791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:08,794 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:08,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39904 deadline: 1679310008794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:08,915 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=428 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/B/6be05936453d46ff92d9f106ac36ad73 2023-03-20 10:59:08,960 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=428 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/C/e0efe4a05a404f868194b15e4b17e8aa 2023-03-20 10:59:08,970 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/A/25593a0c325746188be574597a8934b7 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/25593a0c325746188be574597a8934b7 2023-03-20 10:59:08,975 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/25593a0c325746188be574597a8934b7, entries=150, sequenceid=428, filesize=25.4 K 2023-03-20 10:59:08,976 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/B/6be05936453d46ff92d9f106ac36ad73 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/6be05936453d46ff92d9f106ac36ad73 2023-03-20 10:59:08,980 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/6be05936453d46ff92d9f106ac36ad73, entries=150, sequenceid=428, filesize=11.9 K 2023-03-20 10:59:08,980 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/C/e0efe4a05a404f868194b15e4b17e8aa as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/e0efe4a05a404f868194b15e4b17e8aa 2023-03-20 10:59:08,987 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/e0efe4a05a404f868194b15e4b17e8aa, entries=150, sequenceid=428, filesize=11.9 K 2023-03-20 10:59:08,988 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 96c44321a6ff3c73883afe25fc999ea5 in 935ms, sequenceid=428, compaction requested=true 2023-03-20 10:59:08,988 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 96c44321a6ff3c73883afe25fc999ea5: 2023-03-20 10:59:08,988 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:59:08,988 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:59:08,988 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:59:08,988 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:08,990 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:59:08,990 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:59:08,991 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:08,991 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:08,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:08,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:59:08,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:59:08,991 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:08,991 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:59:08,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:59:08,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:59:08,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:08,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:08,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:59:08,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:59:08,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f5f98ad[Count = 0] remaining members to acquire global barrier 2023-03-20 10:59:08,993 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:59:08,993 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:08,994 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:08,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:08,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:08,994 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:08,994 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:59:08,994 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:59:08,994 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:59:08,994 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:59:08,995 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:08,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:08,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:59:08,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:59:08,995 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:59:08,995 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:59:08,996 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:59:08,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:59:08,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:59:08,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:08,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:08,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:59:08,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:08,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:08,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:59:08,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:59:08,998 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:59:08,998 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:59:08,998 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:59:08,998 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:59:08,998 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:59:08,999 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:08,999 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:59:08,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:08,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:08,999 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:09,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:09,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:59:09,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:59:09,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:59:09,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:59:09,000 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:09,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:59:09,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:09,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:09,001 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:09,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:59:09,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:59:09,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:59:09,005 DEBUG [Thread-3746] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:59:09,005 DEBUG [Thread-3746] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:59:09,004 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:59:09,004 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:09,003 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:59:09,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:59:09,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:59:09,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:59:09,005 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:09,005 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:09,005 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:09,005 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:09,006 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:09,006 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:09,005 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:59:09,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:59:09,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:59:09,006 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:59:09,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 96c44321a6ff3c73883afe25fc999ea5 2023-03-20 10:59:09,098 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 96c44321a6ff3c73883afe25fc999ea5 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-20 10:59:09,098 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 96c44321a6ff3c73883afe25fc999ea5, store=A 2023-03-20 10:59:09,098 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:59:09,098 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 96c44321a6ff3c73883afe25fc999ea5, store=B 2023-03-20 10:59:09,098 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:59:09,098 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 96c44321a6ff3c73883afe25fc999ea5, store=C 2023-03-20 10:59:09,098 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:59:09,105 DEBUG [Thread-3746] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:59:09,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:59:09,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:59:09,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:59:09,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:59:09,109 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:59:09,109 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:59:09,109 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:59:09,109 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:59:09,109 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:09,109 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:09,111 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:59:09,111 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303200a3a711fc7634246923219060ce0adda to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303200a3a711fc7634246923219060ce0adda 2023-03-20 10:59:09,111 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:09,111 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:09,111 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:59:09,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:59:09,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:59:09,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:09,123 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:09,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:59:09,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:09,123 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:59:09,126 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:59:09,126 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:59:09,126 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:59:09,126 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:59:09,126 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:59:09,126 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=442, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/A/a82d245c9c464de28736c2213d44f47d 2023-03-20 10:59:09,126 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:59:09,126 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:59:09,126 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:59:09,126 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. started... 2023-03-20 10:59:09,126 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. as already flushing 2023-03-20 10:59:09,126 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:59:09,126 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:59:09,126 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:59:09,127 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:09,127 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:59:09,127 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:59:09,127 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:09,127 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:09,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:09,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:59:09,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:59:09,128 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:09,128 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:59:09,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:59:09,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:59:09,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:09,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:09,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:59:09,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:59:09,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@504ceb2e[Count = 0] remaining members to acquire global barrier 2023-03-20 10:59:09,130 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:59:09,130 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:09,131 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:09,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:09,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:09,131 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:59:09,131 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:59:09,131 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:59:09,131 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:09,131 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:59:09,132 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:59:09,132 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:59:09,132 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:59:09,132 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:09,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:09,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:59:09,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:59:09,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:59:09,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:59:09,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:09,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:09,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:59:09,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:09,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:09,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:59:09,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:59:09,135 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:59:09,135 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:59:09,135 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:59:09,135 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:59:09,135 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:59:09,135 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=442 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/B/5ef1b438d460497ab4d44b283d89da70 2023-03-20 10:59:09,136 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:09,136 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:59:09,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:09,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:09,136 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:09,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:09,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:59:09,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:59:09,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:59:09,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:59:09,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:09,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:59:09,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:09,137 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:09,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:59:09,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:09,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:09,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:59:09,138 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:09,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:09,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:09,140 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:59:09,140 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:59:09,140 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:09,140 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:59:09,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:59:09,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:59:09,140 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:09,141 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:09,141 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:09,141 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:09,141 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:09,141 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:09,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:59:09,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:59:09,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:59:09,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:59:09,141 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:09,141 DEBUG [Thread-3746] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:59:09,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39798 deadline: 1679310009136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:09,141 DEBUG [Thread-3746] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:59:09,144 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:09,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39904 deadline: 1679310009141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:09,241 DEBUG [Thread-3746] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:59:09,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:59:09,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:59:09,245 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:09,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39798 deadline: 1679310009242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:09,252 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:09,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39904 deadline: 1679310009246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:09,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:59:09,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:59:09,254 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:59:09,254 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:59:09,254 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:59:09,255 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:59:09,256 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:09,257 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:09,257 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:59:09,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:59:09,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:59:09,257 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:09,258 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:09,258 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:59:09,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:09,258 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:09,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:59:09,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:09,258 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:59:09,259 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:59:09,259 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:59:09,259 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:59:09,259 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:59:09,259 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:59:09,259 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:59:09,259 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:59:09,259 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:59:09,259 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. started... 2023-03-20 10:59:09,259 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. as already flushing 2023-03-20 10:59:09,259 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:59:09,259 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:59:09,259 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:59:09,259 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:09,259 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:59:09,260 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:59:09,261 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:09,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:09,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:59:09,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:59:09,261 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:09,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:59:09,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:59:09,262 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:09,262 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:59:09,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:09,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:09,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:59:09,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:59:09,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1943f58f[Count = 0] remaining members to acquire global barrier 2023-03-20 10:59:09,264 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:59:09,264 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:09,264 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:09,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:09,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:09,264 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:59:09,265 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:59:09,265 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:59:09,266 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:59:09,266 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:59:09,266 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:59:09,266 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:09,266 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:59:09,266 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:59:09,266 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:59:09,267 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:59:09,267 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:59:09,267 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:59:09,267 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:59:09,268 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:09,268 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:09,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:09,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:59:09,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:59:09,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:09,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:09,268 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:59:09,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:59:09,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:59:09,268 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:09,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:59:09,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:09,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:09,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:59:09,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:09,272 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:09,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:09,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:59:09,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:09,277 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:59:09,277 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:09,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:59:09,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:59:09,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:59:09,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:59:09,277 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:09,277 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:09,277 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:59:09,277 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:59:09,277 DEBUG [Thread-3746] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:59:09,277 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:59:09,277 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:09,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:59:09,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:59:09,277 DEBUG [Thread-3746] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:59:09,277 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:09,278 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:09,278 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:09,380 DEBUG [Thread-3746] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:59:09,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:59:09,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:59:09,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:59:09,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:59:09,414 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:59:09,414 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:59:09,414 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:59:09,414 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:59:09,415 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:09,415 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:09,424 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:59:09,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:59:09,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:59:09,424 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:09,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:09,427 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:09,427 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:59:09,428 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:09,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:59:09,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:09,428 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:59:09,438 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:59:09,438 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:59:09,438 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:59:09,438 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:59:09,438 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:59:09,439 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:59:09,439 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:59:09,439 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:59:09,439 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. started... 2023-03-20 10:59:09,439 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. as already flushing 2023-03-20 10:59:09,439 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:59:09,439 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:59:09,439 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:59:09,439 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:09,439 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:59:09,439 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:59:09,442 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:09,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:09,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:59:09,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:59:09,442 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:09,442 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:09,442 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:59:09,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:59:09,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:59:09,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:09,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:09,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:59:09,445 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:59:09,446 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ce6dd4d[Count = 0] remaining members to acquire global barrier 2023-03-20 10:59:09,446 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:59:09,446 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:09,446 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:09,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:09,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:09,446 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:59:09,446 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:59:09,446 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:59:09,447 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:09,447 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:59:09,448 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:09,448 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:59:09,448 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:59:09,448 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:59:09,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:09,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:59:09,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:59:09,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:59:09,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:59:09,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:09,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:09,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:59:09,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:09,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:09,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:59:09,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:59:09,451 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:59:09,451 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:59:09,451 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:59:09,451 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:59:09,451 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:59:09,451 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:09,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39798 deadline: 1679310009449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:09,453 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:09,453 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:09,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:09,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:59:09,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:59:09,454 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:59:09,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:09,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:09,456 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:09,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39904 deadline: 1679310009453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:09,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:59:09,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:59:09,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:59:09,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:09,457 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:09,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:09,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:59:09,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:09,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:09,459 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:09,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:59:09,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:09,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:09,469 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:59:09,469 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:59:09,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:59:09,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:59:09,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:59:09,469 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:59:09,469 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:09,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:59:09,469 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:09,469 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:09,469 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:09,469 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:09,469 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:09,470 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:09,470 DEBUG [Thread-3746] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:59:09,470 DEBUG [Thread-3746] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:59:09,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:59:09,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:59:09,544 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=442 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/C/0040a6ce57484d6eb49af9303bb7ca40 2023-03-20 10:59:09,547 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/A/a82d245c9c464de28736c2213d44f47d as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/a82d245c9c464de28736c2213d44f47d 2023-03-20 10:59:09,551 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/a82d245c9c464de28736c2213d44f47d, entries=250, sequenceid=442, filesize=39.2 K 2023-03-20 10:59:09,554 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/B/5ef1b438d460497ab4d44b283d89da70 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/5ef1b438d460497ab4d44b283d89da70 2023-03-20 10:59:09,558 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/5ef1b438d460497ab4d44b283d89da70, entries=150, sequenceid=442, filesize=11.9 K 2023-03-20 10:59:09,561 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/C/0040a6ce57484d6eb49af9303bb7ca40 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/0040a6ce57484d6eb49af9303bb7ca40 2023-03-20 10:59:09,568 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/0040a6ce57484d6eb49af9303bb7ca40, entries=150, sequenceid=442, filesize=11.9 K 2023-03-20 10:59:09,569 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 96c44321a6ff3c73883afe25fc999ea5 in 471ms, sequenceid=442, compaction requested=true 2023-03-20 10:59:09,569 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 96c44321a6ff3c73883afe25fc999ea5: 2023-03-20 10:59:09,569 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:59:09,569 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-03-20 10:59:09,569 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-03-20 10:59:09,569 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:59:09,569 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:59:09,570 DEBUG [Thread-3746] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:59:09,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:59:09,571 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 83506 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-03-20 10:59:09,571 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): 96c44321a6ff3c73883afe25fc999ea5/B is initiating minor compaction (all files) 2023-03-20 10:59:09,571 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 96c44321a6ff3c73883afe25fc999ea5/B in TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:59:09,571 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/b2a986bfe74a48239c2cf212a40504d7, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/55a3321f2c514e2588cfe8d12ffc951a, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/a5892e075c35473c8fd9ef61c2fa0bfd, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/38da7807ab7e460aa877f2d14dc18168, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/3cda0a4a934e47abb159565f9e54c34a, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/6be05936453d46ff92d9f106ac36ad73, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/5ef1b438d460497ab4d44b283d89da70] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp, totalSize=81.5 K 2023-03-20 10:59:09,572 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 189972 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-03-20 10:59:09,572 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting b2a986bfe74a48239c2cf212a40504d7, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=318, earliestPutTs=1679309941984 2023-03-20 10:59:09,572 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1912): 96c44321a6ff3c73883afe25fc999ea5/A is initiating minor compaction (all files) 2023-03-20 10:59:09,572 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 96c44321a6ff3c73883afe25fc999ea5/A in TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:59:09,572 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/eb3a60c80a45422a9ddd115c8b827b71, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/367d20b9632a448291d5e42fb2586fea, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/2372c47596314bb3aea73b00615d948e, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/3a579176ad084af0ba98771ac9317d94, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/d6e0709fdbbb411e821a84f481fc3ed9, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/25593a0c325746188be574597a8934b7, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/a82d245c9c464de28736c2213d44f47d] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp, totalSize=185.5 K 2023-03-20 10:59:09,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:59:09,572 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 55a3321f2c514e2588cfe8d12ffc951a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=346, earliestPutTs=1679309942066 2023-03-20 10:59:09,572 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting eb3a60c80a45422a9ddd115c8b827b71, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=318, earliestPutTs=1679309941984 2023-03-20 10:59:09,572 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting a5892e075c35473c8fd9ef61c2fa0bfd, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=359, earliestPutTs=1679309942712 2023-03-20 10:59:09,573 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 367d20b9632a448291d5e42fb2586fea, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=346, earliestPutTs=1679309942066 2023-03-20 10:59:09,573 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 38da7807ab7e460aa877f2d14dc18168, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=396, earliestPutTs=1679309943947 2023-03-20 10:59:09,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:59:09,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:59:09,573 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:59:09,573 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 2372c47596314bb3aea73b00615d948e, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=359, earliestPutTs=1679309942712 2023-03-20 10:59:09,573 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:59:09,573 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:59:09,573 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:59:09,573 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 3cda0a4a934e47abb159565f9e54c34a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=408, earliestPutTs=1679309945098 2023-03-20 10:59:09,574 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 3a579176ad084af0ba98771ac9317d94, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=396, earliestPutTs=1679309943947 2023-03-20 10:59:09,574 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:09,574 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:09,574 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting d6e0709fdbbb411e821a84f481fc3ed9, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=408, earliestPutTs=1679309945098 2023-03-20 10:59:09,574 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 6be05936453d46ff92d9f106ac36ad73, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=428, earliestPutTs=1679309946304 2023-03-20 10:59:09,575 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting 25593a0c325746188be574597a8934b7, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=428, earliestPutTs=1679309946304 2023-03-20 10:59:09,575 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:59:09,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:59:09,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:59:09,576 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] compactions.Compactor(207): Compacting a82d245c9c464de28736c2213d44f47d, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=442, earliestPutTs=1679309948468 2023-03-20 10:59:09,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:09,576 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:09,576 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:09,576 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:09,576 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:59:09,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:59:09,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:09,576 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:59:09,580 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 5ef1b438d460497ab4d44b283d89da70, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=442, earliestPutTs=1679309948468 2023-03-20 10:59:09,596 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:59:09,597 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:59:09,597 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:59:09,597 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:59:09,597 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:59:09,598 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:59:09,598 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:59:09,598 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:59:09,598 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. started... 2023-03-20 10:59:09,599 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 96c44321a6ff3c73883afe25fc999ea5 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-20 10:59:09,599 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 96c44321a6ff3c73883afe25fc999ea5, store=A 2023-03-20 10:59:09,599 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:59:09,599 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 96c44321a6ff3c73883afe25fc999ea5, store=B 2023-03-20 10:59:09,599 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:59:09,599 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 96c44321a6ff3c73883afe25fc999ea5, store=C 2023-03-20 10:59:09,599 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:59:09,627 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): 96c44321a6ff3c73883afe25fc999ea5#B#compaction#578 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:59:09,639 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] throttle.PressureAwareThroughputController(145): 96c44321a6ff3c73883afe25fc999ea5#A#compaction#577 average throughput is 0.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:59:09,677 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/B/a073bd0afb134e27ba33e386073c242f as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/a073bd0afb134e27ba33e386073c242f 2023-03-20 10:59:09,681 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 96c44321a6ff3c73883afe25fc999ea5/B of 96c44321a6ff3c73883afe25fc999ea5 into a073bd0afb134e27ba33e386073c242f(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:59:09,681 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 96c44321a6ff3c73883afe25fc999ea5: 2023-03-20 10:59:09,681 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5., storeName=96c44321a6ff3c73883afe25fc999ea5/B, priority=9, startTime=1679309949569; duration=0sec 2023-03-20 10:59:09,681 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-20 10:59:09,681 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-03-20 10:59:09,683 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 95590 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-03-20 10:59:09,683 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1912): 96c44321a6ff3c73883afe25fc999ea5/C is initiating minor compaction (all files) 2023-03-20 10:59:09,683 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 96c44321a6ff3c73883afe25fc999ea5/C in TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:59:09,683 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/bd68d316ebc94422b631eb0c304a524d, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/efb22f1cb50f4667883e7fc7fdbae341, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/c4f70332d4624094a04d4d39c2489c4b, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/3134d0ff6c354e81b0627dd51f227a86, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/d38f0fee530c4b7ebecbd4a75dd06446, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/46e69a19113e4249abc471d7779d34f3, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/e0efe4a05a404f868194b15e4b17e8aa, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/0040a6ce57484d6eb49af9303bb7ca40] into tmpdir=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp, totalSize=93.3 K 2023-03-20 10:59:09,684 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting bd68d316ebc94422b631eb0c304a524d, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=307, earliestPutTs=1679309940720 2023-03-20 10:59:09,688 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting efb22f1cb50f4667883e7fc7fdbae341, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=318, earliestPutTs=1679309941984 2023-03-20 10:59:09,688 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting c4f70332d4624094a04d4d39c2489c4b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=346, earliestPutTs=1679309942066 2023-03-20 10:59:09,689 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 3134d0ff6c354e81b0627dd51f227a86, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=359, earliestPutTs=1679309942712 2023-03-20 10:59:09,689 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting d38f0fee530c4b7ebecbd4a75dd06446, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=396, earliestPutTs=1679309943947 2023-03-20 10:59:09,694 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 46e69a19113e4249abc471d7779d34f3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=408, earliestPutTs=1679309945098 2023-03-20 10:59:09,695 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting e0efe4a05a404f868194b15e4b17e8aa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=428, earliestPutTs=1679309946304 2023-03-20 10:59:09,696 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] compactions.Compactor(207): Compacting 0040a6ce57484d6eb49af9303bb7ca40, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=442, earliestPutTs=1679309948468 2023-03-20 10:59:09,716 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] throttle.PressureAwareThroughputController(145): 96c44321a6ff3c73883afe25fc999ea5#C#compaction#580 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-20 10:59:09,716 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/A/485ec7913bd7410a91291c7860661bc6 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/485ec7913bd7410a91291c7860661bc6 2023-03-20 10:59:09,723 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 96c44321a6ff3c73883afe25fc999ea5/A of 96c44321a6ff3c73883afe25fc999ea5 into 485ec7913bd7410a91291c7860661bc6(size=26.3 K), total size for store is 26.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:59:09,723 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 96c44321a6ff3c73883afe25fc999ea5: 2023-03-20 10:59:09,723 INFO [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5., storeName=96c44321a6ff3c73883afe25fc999ea5/A, priority=9, startTime=1679309949569; duration=0sec 2023-03-20 10:59:09,723 DEBUG [RS:0;jenkins-hbase20:36041-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:59:09,731 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/C/77c581d59b8e4806a7a62134d25d3bec as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/77c581d59b8e4806a7a62134d25d3bec 2023-03-20 10:59:09,741 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 96c44321a6ff3c73883afe25fc999ea5/C of 96c44321a6ff3c73883afe25fc999ea5 into 77c581d59b8e4806a7a62134d25d3bec(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-20 10:59:09,741 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 96c44321a6ff3c73883afe25fc999ea5: 2023-03-20 10:59:09,741 INFO [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5., storeName=96c44321a6ff3c73883afe25fc999ea5/C, priority=8, startTime=1679309949569; duration=0sec 2023-03-20 10:59:09,741 DEBUG [RS:0;jenkins-hbase20:36041-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-20 10:59:09,771 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. as already flushing 2023-03-20 10:59:09,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 96c44321a6ff3c73883afe25fc999ea5 2023-03-20 10:59:09,804 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:09,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39904 deadline: 1679310009801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:09,805 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:09,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39798 deadline: 1679310009801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:09,810 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:09,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39816 deadline: 1679310009808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:09,815 DEBUG [Thread-3740] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18268 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5., hostname=jenkins-hbase20.apache.org,36041,1679309777062, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-20 10:59:09,860 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:09,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 148.251.75.209:39832 deadline: 1679310009859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:09,862 DEBUG [Thread-3738] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18317 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5., hostname=jenkins-hbase20.apache.org,36041,1679309777062, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-20 10:59:09,907 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:09,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39798 deadline: 1679310009906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:09,908 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:09,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=36041] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39904 deadline: 1679310009906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:10,088 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230320bd14bcbb80b04b6bac63759fff982e27 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320bd14bcbb80b04b6bac63759fff982e27 2023-03-20 10:59:10,106 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=460, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/A/aa8eb7f100634083b9df6b00179bafdf 2023-03-20 10:59:10,110 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:10,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 304 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39798 deadline: 1679310010109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:10,110 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:10,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39904 deadline: 1679310010110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:10,415 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:10,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 306 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39798 deadline: 1679310010413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:10,415 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-20 10:59:10,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=36041] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 148.251.75.209:39904 deadline: 1679310010413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:10,513 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=460 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/B/f587d6508a294d4e85a3207eb3833e0a 2023-03-20 10:59:10,552 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=460 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/C/8a56f6485aee4a09aded201fbe4b2554 2023-03-20 10:59:10,556 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/A/aa8eb7f100634083b9df6b00179bafdf as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/aa8eb7f100634083b9df6b00179bafdf 2023-03-20 10:59:10,561 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/aa8eb7f100634083b9df6b00179bafdf, entries=150, sequenceid=460, filesize=25.4 K 2023-03-20 10:59:10,561 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/B/f587d6508a294d4e85a3207eb3833e0a as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/f587d6508a294d4e85a3207eb3833e0a 2023-03-20 10:59:10,565 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/f587d6508a294d4e85a3207eb3833e0a, entries=150, sequenceid=460, filesize=11.9 K 2023-03-20 10:59:10,565 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/C/8a56f6485aee4a09aded201fbe4b2554 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/8a56f6485aee4a09aded201fbe4b2554 2023-03-20 10:59:10,569 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/8a56f6485aee4a09aded201fbe4b2554, entries=150, sequenceid=460, filesize=11.9 K 2023-03-20 10:59:10,569 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for 96c44321a6ff3c73883afe25fc999ea5 in 970ms, sequenceid=460, compaction requested=false 2023-03-20 10:59:10,570 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 96c44321a6ff3c73883afe25fc999ea5: 2023-03-20 10:59:10,570 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:59:10,570 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:59:10,570 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:59:10,570 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:10,570 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:59:10,570 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:59:10,571 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:10,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:10,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:59:10,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:59:10,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:59:10,571 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:10,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:59:10,571 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:10,571 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:59:10,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:10,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:10,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:59:10,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:59:10,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f5faa8c[Count = 0] remaining members to acquire global barrier 2023-03-20 10:59:10,572 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:59:10,572 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:10,573 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:10,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:10,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:10,573 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:59:10,573 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:59:10,573 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:10,573 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:59:10,573 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:59:10,574 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:10,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:10,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:59:10,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:59:10,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:59:10,574 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:59:10,574 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:59:10,574 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:59:10,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:59:10,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:10,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:10,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:59:10,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:10,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:10,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:59:10,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:59:10,575 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:59:10,575 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:59:10,575 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:59:10,576 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:59:10,576 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:59:10,576 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:10,576 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:10,576 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:59:10,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:10,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:10,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:59:10,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:59:10,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:10,577 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:10,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:59:10,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:59:10,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:59:10,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:10,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:10,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:59:10,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:10,578 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:10,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:10,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:59:10,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:10,579 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:59:10,579 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:59:10,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:59:10,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:59:10,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:59:10,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:59:10,579 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:59:10,579 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:10,579 DEBUG [Thread-3746] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:59:10,579 DEBUG [Thread-3746] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:59:10,579 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:10,580 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:10,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:59:10,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:59:10,580 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:59:10,580 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:10,580 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:10,580 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:10,580 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:10,680 DEBUG [Thread-3746] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:59:10,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:59:10,682 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(933): Client=jenkins//148.251.75.209 procedure request for: flush-table-proc 2023-03-20 10:59:10,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-20 10:59:10,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-20 10:59:10,689 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-20 10:59:10,689 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:59:10,689 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-20 10:59:10,689 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-20 10:59:10,689 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:10,689 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:10,690 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:59:10,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:59:10,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:59:10,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:10,691 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:10,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-20 10:59:10,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:10,691 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-20 10:59:10,692 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:10,692 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:10,692 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-20 10:59:10,692 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-20 10:59:10,693 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-20 10:59:10,693 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-20 10:59:10,693 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-20 10:59:10,693 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-20 10:59:10,693 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-20 10:59:10,693 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-20 10:59:10,693 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:59:10,693 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. started... 2023-03-20 10:59:10,693 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 96c44321a6ff3c73883afe25fc999ea5 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-20 10:59:10,693 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 96c44321a6ff3c73883afe25fc999ea5, store=A 2023-03-20 10:59:10,693 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:59:10,693 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 96c44321a6ff3c73883afe25fc999ea5, store=B 2023-03-20 10:59:10,693 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:59:10,694 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 96c44321a6ff3c73883afe25fc999ea5, store=C 2023-03-20 10:59:10,694 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:59:10,725 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303200ddf5f213ce440acaeaa747434f9c287 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303200ddf5f213ce440acaeaa747434f9c287 2023-03-20 10:59:10,754 DEBUG [Thread-3749] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x49c34fbe to 127.0.0.1:60226 2023-03-20 10:59:10,754 DEBUG [Thread-3747] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2f6f7187 to 127.0.0.1:60226 2023-03-20 10:59:10,754 DEBUG [Thread-3747] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 10:59:10,754 DEBUG [Thread-3749] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 10:59:10,755 DEBUG [Thread-3753] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x44535519 to 127.0.0.1:60226 2023-03-20 10:59:10,755 DEBUG [Thread-3753] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 10:59:10,757 DEBUG [Thread-3755] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7689435c to 127.0.0.1:60226 2023-03-20 10:59:10,757 DEBUG [Thread-3755] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 10:59:10,757 DEBUG [Thread-3751] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5467be78 to 127.0.0.1:60226 2023-03-20 10:59:10,757 DEBUG [Thread-3751] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 10:59:10,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36041] regionserver.HRegion(9158): Flush requested on 96c44321a6ff3c73883afe25fc999ea5 2023-03-20 10:59:10,919 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. as already flushing 2023-03-20 10:59:10,919 DEBUG [Thread-3744] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7bec0400 to 127.0.0.1:60226 2023-03-20 10:59:10,919 DEBUG [Thread-3744] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 10:59:10,924 DEBUG [Thread-3736] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5ddb3e63 to 127.0.0.1:60226 2023-03-20 10:59:10,924 DEBUG [Thread-3736] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 10:59:11,156 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=478, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/A/11d41d773ad74f19978abb9e8e550d52 2023-03-20 10:59:11,564 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=478 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/B/21784a02e7f24ee3b0d9558c770fe139 2023-03-20 10:59:11,975 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=478 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/C/028f5cf97c77422cb2ccec694fd43ba7 2023-03-20 10:59:11,985 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/A/11d41d773ad74f19978abb9e8e550d52 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/11d41d773ad74f19978abb9e8e550d52 2023-03-20 10:59:11,991 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/11d41d773ad74f19978abb9e8e550d52, entries=150, sequenceid=478, filesize=25.4 K 2023-03-20 10:59:11,992 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/B/21784a02e7f24ee3b0d9558c770fe139 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/21784a02e7f24ee3b0d9558c770fe139 2023-03-20 10:59:12,000 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/21784a02e7f24ee3b0d9558c770fe139, entries=150, sequenceid=478, filesize=11.9 K 2023-03-20 10:59:12,001 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/C/028f5cf97c77422cb2ccec694fd43ba7 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/028f5cf97c77422cb2ccec694fd43ba7 2023-03-20 10:59:12,007 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/028f5cf97c77422cb2ccec694fd43ba7, entries=150, sequenceid=478, filesize=11.9 K 2023-03-20 10:59:12,014 INFO [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=13.42 KB/13740 for 96c44321a6ff3c73883afe25fc999ea5 in 1321ms, sequenceid=478, compaction requested=true 2023-03-20 10:59:12,014 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 96c44321a6ff3c73883afe25fc999ea5: 2023-03-20 10:59:12,014 DEBUG [rs(jenkins-hbase20.apache.org,36041,1679309777062)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:59:12,014 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-20 10:59:12,014 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-20 10:59:12,014 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:12,014 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-20 10:59:12,014 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-20 10:59:12,018 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:12,018 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:12,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:12,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:59:12,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:59:12,018 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:12,018 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-20 10:59:12,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:59:12,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:59:12,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:12,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:12,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:59:12,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase20.apache.org,36041,1679309777062' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-20 10:59:12,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78fd3dd2[Count = 0] remaining members to acquire global barrier 2023-03-20 10:59:12,020 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-20 10:59:12,020 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:12,020 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:12,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:12,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:12,021 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-20 10:59:12,021 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-20 10:59:12,021 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase20.apache.org,36041,1679309777062' in zk 2023-03-20 10:59:12,021 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:12,021 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-20 10:59:12,021 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-20 10:59:12,021 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:59:12,021 DEBUG [member: 'jenkins-hbase20.apache.org,36041,1679309777062' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-20 10:59:12,022 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:12,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:12,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:59:12,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:59:12,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:59:12,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:59:12,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:12,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:12,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:59:12,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:12,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:12,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase20.apache.org,36041,1679309777062': 2023-03-20 10:59:12,024 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase20.apache.org,36041,1679309777062' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-20 10:59:12,024 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-20 10:59:12,024 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-20 10:59:12,024 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-20 10:59:12,024 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-20 10:59:12,024 INFO [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-20 10:59:12,026 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:12,026 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:12,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:12,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:12,026 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:59:12,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:12,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-20 10:59:12,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-20 10:59:12,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:59:12,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:59:12,026 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:12,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-20 10:59:12,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:12,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:12,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-20 10:59:12,027 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:12,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:12,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:12,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-20 10:59:12,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-20 10:59:12,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:12,032 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:12,032 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-20 10:59:12,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-20 10:59:12,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-20 10:59:12,032 DEBUG [(jenkins-hbase20.apache.org,44257,1679309775907)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-20 10:59:12,032 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:12,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-20 10:59:12,032 DEBUG [Thread-3746] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-20 10:59:12,032 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-20 10:59:12,032 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-20 10:59:12,032 DEBUG [Thread-3746] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-20 10:59:12,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-20 10:59:12,032 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-20 10:59:12,032 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:12,032 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:12,032 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-20 10:59:12,032 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-20 10:59:12,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-20 10:59:12,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-20 10:59:12,132 DEBUG [Thread-3746] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-20 10:59:12,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-20 10:59:14,193 DEBUG [Thread-3742] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1ae3ec16 to 127.0.0.1:60226 2023-03-20 10:59:14,193 DEBUG [Thread-3742] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 10:59:19,079 DEBUG [regionserver/jenkins-hbase20:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-03-20 10:59:19,830 DEBUG [Thread-3740] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5ec98d63 to 127.0.0.1:60226 2023-03-20 10:59:19,830 DEBUG [Thread-3740] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 10:59:19,934 DEBUG [Thread-3738] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x630ea023 to 127.0.0.1:60226 2023-03-20 10:59:19,934 DEBUG [Thread-3738] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 10:59:19,934 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-20 10:59:19,935 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(394): wrote 147 2023-03-20 10:59:19,935 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(394): wrote 15 2023-03-20 10:59:19,935 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(394): wrote 16 2023-03-20 10:59:19,935 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(394): wrote 42 2023-03-20 10:59:19,935 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(394): wrote 175 2023-03-20 10:59:19,935 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-20 10:59:19,935 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(398): read 2893 2023-03-20 10:59:19,935 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(398): read 2918 2023-03-20 10:59:19,935 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(398): read 2900 2023-03-20 10:59:19,935 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(398): read 2901 2023-03-20 10:59:19,935 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(398): read 2886 2023-03-20 10:59:19,935 INFO [Listener at localhost.localdomain/43509] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-20 10:59:19,935 INFO [Listener at localhost.localdomain/43509] client.ConnectionImplementation(1959): Closing master protocol: MasterService 2023-03-20 10:59:19,935 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4ae8f5b1 to 127.0.0.1:60226 2023-03-20 10:59:19,935 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 10:59:19,936 INFO [Listener at localhost.localdomain/43509] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-20 10:59:19,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.HMaster$11(2418): Client=jenkins//148.251.75.209 disable TestAcidGuarantees 2023-03-20 10:59:19,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-20 10:59:19,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-20 10:59:19,940 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679309959940"}]},"ts":"1679309959940"} 2023-03-20 10:59:19,951 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-20 10:59:19,952 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-20 10:59:19,953 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=96c44321a6ff3c73883afe25fc999ea5, UNASSIGN}] 2023-03-20 10:59:19,954 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=96c44321a6ff3c73883afe25fc999ea5, UNASSIGN 2023-03-20 10:59:19,955 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=96c44321a6ff3c73883afe25fc999ea5, regionState=CLOSING, regionLocation=jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:19,955 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679309959955"},{"qualifier":"sn","vlen":46,"tag":[],"timestamp":"1679309959955"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679309959955"}]},"ts":"1679309959955"} 2023-03-20 10:59:19,957 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE; CloseRegionProcedure 96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062}] 2023-03-20 10:59:20,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-20 10:59:20,110 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(109): Close 96c44321a6ff3c73883afe25fc999ea5 2023-03-20 10:59:20,111 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 96c44321a6ff3c73883afe25fc999ea5, disabling compactions & flushes 2023-03-20 10:59:20,111 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:59:20,111 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:59:20,111 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. after waiting 0 ms 2023-03-20 10:59:20,111 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:59:20,111 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 96c44321a6ff3c73883afe25fc999ea5 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-03-20 10:59:20,111 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 96c44321a6ff3c73883afe25fc999ea5, store=A 2023-03-20 10:59:20,111 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:59:20,111 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 96c44321a6ff3c73883afe25fc999ea5, store=B 2023-03-20 10:59:20,111 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:59:20,111 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 96c44321a6ff3c73883afe25fc999ea5, store=C 2023-03-20 10:59:20,111 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-20 10:59:20,125 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230320653e65db6a8a4405941d71300f1a0c1b to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320653e65db6a8a4405941d71300f1a0c1b 2023-03-20 10:59:20,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-20 10:59:20,536 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=486, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/A/72e402d636b049ae935a154ac7ed73a1 2023-03-20 10:59:20,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-20 10:59:20,547 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=486 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/B/2def7b42a1864729aea3f7180e280df6 2023-03-20 10:59:20,963 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=486 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/C/8c463cde384d415aac6a006fcd42f188 2023-03-20 10:59:20,966 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/A/72e402d636b049ae935a154ac7ed73a1 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/72e402d636b049ae935a154ac7ed73a1 2023-03-20 10:59:20,969 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/72e402d636b049ae935a154ac7ed73a1, entries=100, sequenceid=486, filesize=18.6 K 2023-03-20 10:59:20,970 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/B/2def7b42a1864729aea3f7180e280df6 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/2def7b42a1864729aea3f7180e280df6 2023-03-20 10:59:20,976 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/2def7b42a1864729aea3f7180e280df6, entries=100, sequenceid=486, filesize=9.5 K 2023-03-20 10:59:20,976 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/.tmp/C/8c463cde384d415aac6a006fcd42f188 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/8c463cde384d415aac6a006fcd42f188 2023-03-20 10:59:20,983 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/8c463cde384d415aac6a006fcd42f188, entries=100, sequenceid=486, filesize=9.5 K 2023-03-20 10:59:20,983 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for 96c44321a6ff3c73883afe25fc999ea5 in 872ms, sequenceid=486, compaction requested=true 2023-03-20 10:59:21,002 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/a0e8d5a34c684ebca98a224c039dce91, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/7ae3e523a54e4b27bcb77bdb6a21c577, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/d17893d5991a4ae2a44caec61d1385ca, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/e35e127666fd416e9a8ab32872b47db4, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/56a19abb20c841a18893eed3c7781d1a, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/576b677cd9ad4a5b8cfcc97aea28753a, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/7159cdee5fa44dadae506246e1138b1b, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/d3b5109d762340c5a1efa2f613c073ee, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/6e7dcd9e6c99421fb9ba4903fe52b2e6, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/e7304005dee84e6ca4f434b375db2f57, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/737085474471476bba636cc9ff3f15d9, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/2ff805f7b93b4c0b9a10bf3e40dd789d, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/3062807056ac4d6db02049608bda4853, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/fda06680636e4a7680027a86919db163, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/9cbb4507d344426eb398e5ee4891374e, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/9931de83d4eb48b1afbfe38048254fb0, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/719dd91fcc1b4e69a3cbf6cae3a9b181, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/58ad4501bfb54a8d8935bb8be9fc556a, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/eb3a60c80a45422a9ddd115c8b827b71, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/367d20b9632a448291d5e42fb2586fea, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/2372c47596314bb3aea73b00615d948e, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/3a579176ad084af0ba98771ac9317d94, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/d6e0709fdbbb411e821a84f481fc3ed9, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/25593a0c325746188be574597a8934b7, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/a82d245c9c464de28736c2213d44f47d] to archive 2023-03-20 10:59:21,003 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-20 10:59:21,004 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/a0e8d5a34c684ebca98a224c039dce91 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/a0e8d5a34c684ebca98a224c039dce91 2023-03-20 10:59:21,005 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/7ae3e523a54e4b27bcb77bdb6a21c577 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/7ae3e523a54e4b27bcb77bdb6a21c577 2023-03-20 10:59:21,005 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/d17893d5991a4ae2a44caec61d1385ca to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/d17893d5991a4ae2a44caec61d1385ca 2023-03-20 10:59:21,006 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/e35e127666fd416e9a8ab32872b47db4 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/e35e127666fd416e9a8ab32872b47db4 2023-03-20 10:59:21,007 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/56a19abb20c841a18893eed3c7781d1a to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/56a19abb20c841a18893eed3c7781d1a 2023-03-20 10:59:21,007 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/576b677cd9ad4a5b8cfcc97aea28753a to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/576b677cd9ad4a5b8cfcc97aea28753a 2023-03-20 10:59:21,008 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/7159cdee5fa44dadae506246e1138b1b to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/7159cdee5fa44dadae506246e1138b1b 2023-03-20 10:59:21,009 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/d3b5109d762340c5a1efa2f613c073ee to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/d3b5109d762340c5a1efa2f613c073ee 2023-03-20 10:59:21,010 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/6e7dcd9e6c99421fb9ba4903fe52b2e6 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/6e7dcd9e6c99421fb9ba4903fe52b2e6 2023-03-20 10:59:21,011 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/e7304005dee84e6ca4f434b375db2f57 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/e7304005dee84e6ca4f434b375db2f57 2023-03-20 10:59:21,012 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/737085474471476bba636cc9ff3f15d9 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/737085474471476bba636cc9ff3f15d9 2023-03-20 10:59:21,013 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/2ff805f7b93b4c0b9a10bf3e40dd789d to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/2ff805f7b93b4c0b9a10bf3e40dd789d 2023-03-20 10:59:21,014 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/3062807056ac4d6db02049608bda4853 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/3062807056ac4d6db02049608bda4853 2023-03-20 10:59:21,015 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/fda06680636e4a7680027a86919db163 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/fda06680636e4a7680027a86919db163 2023-03-20 10:59:21,016 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/9cbb4507d344426eb398e5ee4891374e to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/9cbb4507d344426eb398e5ee4891374e 2023-03-20 10:59:21,023 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/9931de83d4eb48b1afbfe38048254fb0 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/9931de83d4eb48b1afbfe38048254fb0 2023-03-20 10:59:21,024 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/719dd91fcc1b4e69a3cbf6cae3a9b181 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/719dd91fcc1b4e69a3cbf6cae3a9b181 2023-03-20 10:59:21,025 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/58ad4501bfb54a8d8935bb8be9fc556a to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/58ad4501bfb54a8d8935bb8be9fc556a 2023-03-20 10:59:21,026 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/eb3a60c80a45422a9ddd115c8b827b71 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/eb3a60c80a45422a9ddd115c8b827b71 2023-03-20 10:59:21,029 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/367d20b9632a448291d5e42fb2586fea to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/367d20b9632a448291d5e42fb2586fea 2023-03-20 10:59:21,030 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/2372c47596314bb3aea73b00615d948e to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/2372c47596314bb3aea73b00615d948e 2023-03-20 10:59:21,031 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/3a579176ad084af0ba98771ac9317d94 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/3a579176ad084af0ba98771ac9317d94 2023-03-20 10:59:21,032 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/d6e0709fdbbb411e821a84f481fc3ed9 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/d6e0709fdbbb411e821a84f481fc3ed9 2023-03-20 10:59:21,033 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/25593a0c325746188be574597a8934b7 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/25593a0c325746188be574597a8934b7 2023-03-20 10:59:21,034 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/a82d245c9c464de28736c2213d44f47d to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/a82d245c9c464de28736c2213d44f47d 2023-03-20 10:59:21,037 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/cef3dd738cbe455aa74b26b8ef844e79, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/537ae667411d48bda05589632378b39b, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/ac5c7965469f42a7b932280c5c57d2fa, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/e943679fc0494b84b27bee7dfcff505c, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/8358210835f8404f9d814c6f0867bebe, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/a3547e72f05c4630a819691105b83e3b, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/f7183f8e8bb9493b9855a9cf2a36f32c, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/dc578bc354c3476a886b66bb9a0e6b5b, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/e73c6c855a134523855c18812a8cfd88, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/baf414c3fd434b96a71c8fcac54370b6, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/e8b702252a014fb0a0030659ae17e1c1, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/78dd247c21d7478eb666bc71a20cdf05, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/653e7e82f9e148b3b0c275e5de0d5427, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/5ce25c323bad4bf3afe9cef379845c3f, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/18918dd42766411f90a8b8fd50500f3c, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/b007d2fddf404bba91c71abb80450208, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/9613b23480154d6e8bc8a8e0adf5fed1, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/bab1ea6e6840444fbd02d25522d7a7b6, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/b2a986bfe74a48239c2cf212a40504d7, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/4509c2b8da064dc9a6c83c2f3ce4f0dd, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/55a3321f2c514e2588cfe8d12ffc951a, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/a5892e075c35473c8fd9ef61c2fa0bfd, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/38da7807ab7e460aa877f2d14dc18168, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/3cda0a4a934e47abb159565f9e54c34a, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/6be05936453d46ff92d9f106ac36ad73, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/5ef1b438d460497ab4d44b283d89da70] to archive 2023-03-20 10:59:21,038 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-20 10:59:21,040 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/cef3dd738cbe455aa74b26b8ef844e79 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/cef3dd738cbe455aa74b26b8ef844e79 2023-03-20 10:59:21,041 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/537ae667411d48bda05589632378b39b to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/537ae667411d48bda05589632378b39b 2023-03-20 10:59:21,042 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/ac5c7965469f42a7b932280c5c57d2fa to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/ac5c7965469f42a7b932280c5c57d2fa 2023-03-20 10:59:21,043 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/e943679fc0494b84b27bee7dfcff505c to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/e943679fc0494b84b27bee7dfcff505c 2023-03-20 10:59:21,044 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/8358210835f8404f9d814c6f0867bebe to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/8358210835f8404f9d814c6f0867bebe 2023-03-20 10:59:21,045 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/a3547e72f05c4630a819691105b83e3b to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/a3547e72f05c4630a819691105b83e3b 2023-03-20 10:59:21,046 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/f7183f8e8bb9493b9855a9cf2a36f32c to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/f7183f8e8bb9493b9855a9cf2a36f32c 2023-03-20 10:59:21,047 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/dc578bc354c3476a886b66bb9a0e6b5b to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/dc578bc354c3476a886b66bb9a0e6b5b 2023-03-20 10:59:21,048 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/e73c6c855a134523855c18812a8cfd88 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/e73c6c855a134523855c18812a8cfd88 2023-03-20 10:59:21,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-20 10:59:21,049 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/baf414c3fd434b96a71c8fcac54370b6 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/baf414c3fd434b96a71c8fcac54370b6 2023-03-20 10:59:21,050 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/e8b702252a014fb0a0030659ae17e1c1 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/e8b702252a014fb0a0030659ae17e1c1 2023-03-20 10:59:21,050 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/78dd247c21d7478eb666bc71a20cdf05 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/78dd247c21d7478eb666bc71a20cdf05 2023-03-20 10:59:21,051 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/653e7e82f9e148b3b0c275e5de0d5427 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/653e7e82f9e148b3b0c275e5de0d5427 2023-03-20 10:59:21,052 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/5ce25c323bad4bf3afe9cef379845c3f to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/5ce25c323bad4bf3afe9cef379845c3f 2023-03-20 10:59:21,053 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/18918dd42766411f90a8b8fd50500f3c to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/18918dd42766411f90a8b8fd50500f3c 2023-03-20 10:59:21,053 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/b007d2fddf404bba91c71abb80450208 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/b007d2fddf404bba91c71abb80450208 2023-03-20 10:59:21,054 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/9613b23480154d6e8bc8a8e0adf5fed1 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/9613b23480154d6e8bc8a8e0adf5fed1 2023-03-20 10:59:21,055 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/bab1ea6e6840444fbd02d25522d7a7b6 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/bab1ea6e6840444fbd02d25522d7a7b6 2023-03-20 10:59:21,056 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/b2a986bfe74a48239c2cf212a40504d7 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/b2a986bfe74a48239c2cf212a40504d7 2023-03-20 10:59:21,057 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/4509c2b8da064dc9a6c83c2f3ce4f0dd to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/4509c2b8da064dc9a6c83c2f3ce4f0dd 2023-03-20 10:59:21,058 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/55a3321f2c514e2588cfe8d12ffc951a to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/55a3321f2c514e2588cfe8d12ffc951a 2023-03-20 10:59:21,059 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/a5892e075c35473c8fd9ef61c2fa0bfd to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/a5892e075c35473c8fd9ef61c2fa0bfd 2023-03-20 10:59:21,060 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/38da7807ab7e460aa877f2d14dc18168 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/38da7807ab7e460aa877f2d14dc18168 2023-03-20 10:59:21,060 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/3cda0a4a934e47abb159565f9e54c34a to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/3cda0a4a934e47abb159565f9e54c34a 2023-03-20 10:59:21,061 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/6be05936453d46ff92d9f106ac36ad73 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/6be05936453d46ff92d9f106ac36ad73 2023-03-20 10:59:21,062 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/5ef1b438d460497ab4d44b283d89da70 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/5ef1b438d460497ab4d44b283d89da70 2023-03-20 10:59:21,065 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/5753951e6e9743b3a97400013dadc78c, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/7c50fcf180334f9aae90b0d6a456cfe5, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/349b1558bd3e44d3b8fa0359525c479d, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/d819f9d6802646d89c280a75b5e4e7fa, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/89696bd472024b7f90144851f8723817, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/5f09e1bf251e427ba28d11621de7bc6a, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/d6ad15fa78c64850abe8bb71ab916eb3, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/ffd0cbd4bef0470d86e63124fe0923f8, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/5ec49e314c164d7dbaa75f63f02328ae, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/43bbc546850743e88c90457a64cbc411, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/639059aa57ab48fe8037cc95786413b4, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/3fbf0c8c0d324c47a7daa80b2a593e8d, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/5cf3d45f8d6f4612b67f81f25e543d64, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/a01c3fdcd28c48d9855ba61d4957b4cd, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/6e9f1474590f4a68a3e932991ed4d71b, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/bd68d316ebc94422b631eb0c304a524d, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/e75db226afe346edb9e1595284687883, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/efb22f1cb50f4667883e7fc7fdbae341, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/c4f70332d4624094a04d4d39c2489c4b, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/3134d0ff6c354e81b0627dd51f227a86, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/d38f0fee530c4b7ebecbd4a75dd06446, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/46e69a19113e4249abc471d7779d34f3, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/e0efe4a05a404f868194b15e4b17e8aa, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/0040a6ce57484d6eb49af9303bb7ca40] to archive 2023-03-20 10:59:21,070 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-20 10:59:21,072 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/5753951e6e9743b3a97400013dadc78c to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/5753951e6e9743b3a97400013dadc78c 2023-03-20 10:59:21,073 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/7c50fcf180334f9aae90b0d6a456cfe5 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/7c50fcf180334f9aae90b0d6a456cfe5 2023-03-20 10:59:21,074 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/349b1558bd3e44d3b8fa0359525c479d to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/349b1558bd3e44d3b8fa0359525c479d 2023-03-20 10:59:21,075 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/d819f9d6802646d89c280a75b5e4e7fa to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/d819f9d6802646d89c280a75b5e4e7fa 2023-03-20 10:59:21,076 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/89696bd472024b7f90144851f8723817 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/89696bd472024b7f90144851f8723817 2023-03-20 10:59:21,077 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/5f09e1bf251e427ba28d11621de7bc6a to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/5f09e1bf251e427ba28d11621de7bc6a 2023-03-20 10:59:21,078 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/d6ad15fa78c64850abe8bb71ab916eb3 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/d6ad15fa78c64850abe8bb71ab916eb3 2023-03-20 10:59:21,079 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/ffd0cbd4bef0470d86e63124fe0923f8 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/ffd0cbd4bef0470d86e63124fe0923f8 2023-03-20 10:59:21,080 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/5ec49e314c164d7dbaa75f63f02328ae to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/5ec49e314c164d7dbaa75f63f02328ae 2023-03-20 10:59:21,080 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/43bbc546850743e88c90457a64cbc411 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/43bbc546850743e88c90457a64cbc411 2023-03-20 10:59:21,081 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/639059aa57ab48fe8037cc95786413b4 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/639059aa57ab48fe8037cc95786413b4 2023-03-20 10:59:21,082 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/3fbf0c8c0d324c47a7daa80b2a593e8d to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/3fbf0c8c0d324c47a7daa80b2a593e8d 2023-03-20 10:59:21,083 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/5cf3d45f8d6f4612b67f81f25e543d64 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/5cf3d45f8d6f4612b67f81f25e543d64 2023-03-20 10:59:21,084 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/a01c3fdcd28c48d9855ba61d4957b4cd to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/a01c3fdcd28c48d9855ba61d4957b4cd 2023-03-20 10:59:21,086 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/6e9f1474590f4a68a3e932991ed4d71b to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/6e9f1474590f4a68a3e932991ed4d71b 2023-03-20 10:59:21,087 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/bd68d316ebc94422b631eb0c304a524d to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/bd68d316ebc94422b631eb0c304a524d 2023-03-20 10:59:21,089 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/e75db226afe346edb9e1595284687883 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/e75db226afe346edb9e1595284687883 2023-03-20 10:59:21,090 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/efb22f1cb50f4667883e7fc7fdbae341 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/efb22f1cb50f4667883e7fc7fdbae341 2023-03-20 10:59:21,092 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/c4f70332d4624094a04d4d39c2489c4b to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/c4f70332d4624094a04d4d39c2489c4b 2023-03-20 10:59:21,093 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/3134d0ff6c354e81b0627dd51f227a86 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/3134d0ff6c354e81b0627dd51f227a86 2023-03-20 10:59:21,096 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/d38f0fee530c4b7ebecbd4a75dd06446 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/d38f0fee530c4b7ebecbd4a75dd06446 2023-03-20 10:59:21,098 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/46e69a19113e4249abc471d7779d34f3 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/46e69a19113e4249abc471d7779d34f3 2023-03-20 10:59:21,099 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/e0efe4a05a404f868194b15e4b17e8aa to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/e0efe4a05a404f868194b15e4b17e8aa 2023-03-20 10:59:21,100 DEBUG [StoreCloser-TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/0040a6ce57484d6eb49af9303bb7ca40 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/0040a6ce57484d6eb49af9303bb7ca40 2023-03-20 10:59:21,105 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/recovered.edits/489.seqid, newMaxSeqId=489, maxSeqId=4 2023-03-20 10:59:21,107 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5. 2023-03-20 10:59:21,107 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 96c44321a6ff3c73883afe25fc999ea5: 2023-03-20 10:59:21,108 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.UnassignRegionHandler(147): Closed 96c44321a6ff3c73883afe25fc999ea5 2023-03-20 10:59:21,109 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=96c44321a6ff3c73883afe25fc999ea5, regionState=CLOSED 2023-03-20 10:59:21,109 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679309961109"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679309961109"}]},"ts":"1679309961109"} 2023-03-20 10:59:21,111 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-03-20 10:59:21,111 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; CloseRegionProcedure 96c44321a6ff3c73883afe25fc999ea5, server=jenkins-hbase20.apache.org,36041,1679309777062 in 1.1530 sec 2023-03-20 10:59:21,113 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=63, resume processing ppid=62 2023-03-20 10:59:21,113 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=63, ppid=62, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=96c44321a6ff3c73883afe25fc999ea5, UNASSIGN in 1.1580 sec 2023-03-20 10:59:21,113 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679309961113"}]},"ts":"1679309961113"} 2023-03-20 10:59:21,114 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-20 10:59:21,115 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-20 10:59:21,117 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.1800 sec 2023-03-20 10:59:22,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-20 10:59:22,049 INFO [Listener at localhost.localdomain/43509] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 62 completed 2023-03-20 10:59:22,050 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.HMaster$5(2228): Client=jenkins//148.251.75.209 delete TestAcidGuarantees 2023-03-20 10:59:22,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] procedure2.ProcedureExecutor(1029): Stored pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-20 10:59:22,056 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-20 10:59:22,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-03-20 10:59:22,057 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=65, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-20 10:59:22,062 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5 2023-03-20 10:59:22,065 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A, FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B, FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C, FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/recovered.edits] 2023-03-20 10:59:22,068 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/11d41d773ad74f19978abb9e8e550d52 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/11d41d773ad74f19978abb9e8e550d52 2023-03-20 10:59:22,069 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/485ec7913bd7410a91291c7860661bc6 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/485ec7913bd7410a91291c7860661bc6 2023-03-20 10:59:22,071 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/72e402d636b049ae935a154ac7ed73a1 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/72e402d636b049ae935a154ac7ed73a1 2023-03-20 10:59:22,071 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/aa8eb7f100634083b9df6b00179bafdf to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/A/aa8eb7f100634083b9df6b00179bafdf 2023-03-20 10:59:22,074 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/21784a02e7f24ee3b0d9558c770fe139 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/21784a02e7f24ee3b0d9558c770fe139 2023-03-20 10:59:22,076 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/2def7b42a1864729aea3f7180e280df6 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/2def7b42a1864729aea3f7180e280df6 2023-03-20 10:59:22,077 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/a073bd0afb134e27ba33e386073c242f to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/a073bd0afb134e27ba33e386073c242f 2023-03-20 10:59:22,078 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/f587d6508a294d4e85a3207eb3833e0a to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/B/f587d6508a294d4e85a3207eb3833e0a 2023-03-20 10:59:22,080 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/028f5cf97c77422cb2ccec694fd43ba7 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/028f5cf97c77422cb2ccec694fd43ba7 2023-03-20 10:59:22,081 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/77c581d59b8e4806a7a62134d25d3bec to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/77c581d59b8e4806a7a62134d25d3bec 2023-03-20 10:59:22,081 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/8a56f6485aee4a09aded201fbe4b2554 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/8a56f6485aee4a09aded201fbe4b2554 2023-03-20 10:59:22,084 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/8c463cde384d415aac6a006fcd42f188 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/C/8c463cde384d415aac6a006fcd42f188 2023-03-20 10:59:22,086 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/recovered.edits/489.seqid to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5/recovered.edits/489.seqid 2023-03-20 10:59:22,087 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/.tmp/data/default/TestAcidGuarantees/96c44321a6ff3c73883afe25fc999ea5 2023-03-20 10:59:22,087 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-20 10:59:22,087 DEBUG [PEWorker-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-20 10:59:22,088 DEBUG [PEWorker-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-03-20 10:59:22,090 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303200a3a711fc7634246923219060ce0adda to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303200a3a711fc7634246923219060ce0adda 2023-03-20 10:59:22,091 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303200ddf5f213ce440acaeaa747434f9c287 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303200ddf5f213ce440acaeaa747434f9c287 2023-03-20 10:59:22,095 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303201085b13b9ce34cc59289601f30ded8a9 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303201085b13b9ce34cc59289601f30ded8a9 2023-03-20 10:59:22,096 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303201ba4512296df40d68294e8cb4b2ba8be to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303201ba4512296df40d68294e8cb4b2ba8be 2023-03-20 10:59:22,097 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032025cbb804680a4b9e9a04a7ab57855d6e to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032025cbb804680a4b9e9a04a7ab57855d6e 2023-03-20 10:59:22,098 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032031e9312f50504ccd925f32a6bbe1cc37 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032031e9312f50504ccd925f32a6bbe1cc37 2023-03-20 10:59:22,099 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303204677950d58c34e29be5b187ec5b529bc to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303204677950d58c34e29be5b187ec5b529bc 2023-03-20 10:59:22,100 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303204ba1bcc562b9435ca34124cc47f41df2 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303204ba1bcc562b9435ca34124cc47f41df2 2023-03-20 10:59:22,101 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303205df02a23a2ee4e49a611493be1b7a4ac to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303205df02a23a2ee4e49a611493be1b7a4ac 2023-03-20 10:59:22,102 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320653e65db6a8a4405941d71300f1a0c1b to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320653e65db6a8a4405941d71300f1a0c1b 2023-03-20 10:59:22,103 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303206964d499908744ee895fcf8fd273509f to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303206964d499908744ee895fcf8fd273509f 2023-03-20 10:59:22,104 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303206da75ce969684a6b8cdaa5f79aa88149 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303206da75ce969684a6b8cdaa5f79aa88149 2023-03-20 10:59:22,104 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303208888edf6d56b4b7e8ed63f886a504f96 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303208888edf6d56b4b7e8ed63f886a504f96 2023-03-20 10:59:22,105 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032089077b5d20ad4d82a65ce62763212b17 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032089077b5d20ad4d82a65ce62763212b17 2023-03-20 10:59:22,106 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320af1aef6144754525aa70331acdd5a5a4 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320af1aef6144754525aa70331acdd5a5a4 2023-03-20 10:59:22,107 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320b6979c8e9357488aa74ad04b575d22c0 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320b6979c8e9357488aa74ad04b575d22c0 2023-03-20 10:59:22,107 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320b72d2372c9084bc58a6fc03dd3487477 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320b72d2372c9084bc58a6fc03dd3487477 2023-03-20 10:59:22,108 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320bd14bcbb80b04b6bac63759fff982e27 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320bd14bcbb80b04b6bac63759fff982e27 2023-03-20 10:59:22,109 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320c238bc5caef1436d8c125abc2499a078 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320c238bc5caef1436d8c125abc2499a078 2023-03-20 10:59:22,111 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320c5446a772d384e659268ef93b3aa47de to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320c5446a772d384e659268ef93b3aa47de 2023-03-20 10:59:22,112 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320c863779ab15945ab9ca28748e7d8ab72 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320c863779ab15945ab9ca28748e7d8ab72 2023-03-20 10:59:22,112 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320cecb3b6108844f86a77d3bd285ae34c1 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320cecb3b6108844f86a77d3bd285ae34c1 2023-03-20 10:59:22,113 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320cefc6a6e7a9e42c0a10fb0d03809cf43 to hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230320cefc6a6e7a9e42c0a10fb0d03809cf43 2023-03-20 10:59:22,114 DEBUG [PEWorker-5] backup.HFileArchiver(596): Deleted hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-20 10:59:22,116 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=65, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-20 10:59:22,117 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-20 10:59:22,118 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-20 10:59:22,119 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=65, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-20 10:59:22,119 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-20 10:59:22,120 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1679309962119"}]},"ts":"9223372036854775807"} 2023-03-20 10:59:22,121 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-20 10:59:22,121 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 96c44321a6ff3c73883afe25fc999ea5, NAME => 'TestAcidGuarantees,,1679309928556.96c44321a6ff3c73883afe25fc999ea5.', STARTKEY => '', ENDKEY => ''}] 2023-03-20 10:59:22,121 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-20 10:59:22,121 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1679309962121"}]},"ts":"9223372036854775807"} 2023-03-20 10:59:22,126 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-20 10:59:22,128 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=65, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-20 10:59:22,129 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=65, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 78 msec 2023-03-20 10:59:22,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=44257] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-03-20 10:59:22,158 INFO [Listener at localhost.localdomain/43509] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 65 completed 2023-03-20 10:59:22,171 INFO [Listener at localhost.localdomain/43509] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=225 (was 225), OpenFileDescriptor=516 (was 551), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=756 (was 717) - SystemLoadAverage LEAK? -, ProcessCount=184 (was 184), AvailableMemoryMB=9529 (was 9054) - AvailableMemoryMB LEAK? - 2023-03-20 10:59:22,171 INFO [Listener at localhost.localdomain/43509] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-03-20 10:59:22,171 INFO [Listener at localhost.localdomain/43509] client.ConnectionImplementation(1959): Closing master protocol: MasterService 2023-03-20 10:59:22,171 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x70531455 to 127.0.0.1:60226 2023-03-20 10:59:22,172 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 10:59:22,172 DEBUG [Listener at localhost.localdomain/43509] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-03-20 10:59:22,172 DEBUG [Listener at localhost.localdomain/43509] util.JVMClusterUtil(257): Found active master hash=1208775324, stopped=false 2023-03-20 10:59:22,172 INFO [Listener at localhost.localdomain/43509] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase20.apache.org,44257,1679309775907 2023-03-20 10:59:22,173 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-03-20 10:59:22,173 INFO [Listener at localhost.localdomain/43509] procedure2.ProcedureExecutor(629): Stopping 2023-03-20 10:59:22,173 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-03-20 10:59:22,173 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-20 10:59:22,177 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-20 10:59:22,177 DEBUG [Listener at localhost.localdomain/43509] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x58fe3863 to 127.0.0.1:60226 2023-03-20 10:59:22,177 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-20 10:59:22,177 DEBUG [Listener at localhost.localdomain/43509] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 10:59:22,178 INFO [Listener at localhost.localdomain/43509] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase20.apache.org,36041,1679309777062' ***** 2023-03-20 10:59:22,178 INFO [Listener at localhost.localdomain/43509] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-03-20 10:59:22,178 INFO [RS:0;jenkins-hbase20:36041] regionserver.HeapMemoryManager(220): Stopping 2023-03-20 10:59:22,178 INFO [RS:0;jenkins-hbase20:36041] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-03-20 10:59:22,178 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-03-20 10:59:22,178 INFO [RS:0;jenkins-hbase20:36041] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-03-20 10:59:22,178 INFO [RS:0;jenkins-hbase20:36041] regionserver.HRegionServer(3304): Received CLOSE for 39b9d351fb81939da79179eb526f8a01 2023-03-20 10:59:22,179 INFO [RS:0;jenkins-hbase20:36041] regionserver.HRegionServer(1145): stopping server jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:22,180 DEBUG [RS:0;jenkins-hbase20:36041] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x57f52db3 to 127.0.0.1:60226 2023-03-20 10:59:22,180 DEBUG [RS:0;jenkins-hbase20:36041] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 10:59:22,180 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 39b9d351fb81939da79179eb526f8a01, disabling compactions & flushes 2023-03-20 10:59:22,180 INFO [RS:0;jenkins-hbase20:36041] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-03-20 10:59:22,180 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1679309780495.39b9d351fb81939da79179eb526f8a01. 2023-03-20 10:59:22,180 INFO [RS:0;jenkins-hbase20:36041] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-03-20 10:59:22,181 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1679309780495.39b9d351fb81939da79179eb526f8a01. 2023-03-20 10:59:22,181 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1679309780495.39b9d351fb81939da79179eb526f8a01. after waiting 0 ms 2023-03-20 10:59:22,181 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1679309780495.39b9d351fb81939da79179eb526f8a01. 2023-03-20 10:59:22,181 INFO [RS:0;jenkins-hbase20:36041] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-03-20 10:59:22,181 INFO [RS:0;jenkins-hbase20:36041] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-03-20 10:59:22,181 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 39b9d351fb81939da79179eb526f8a01 1/1 column families, dataSize=78 B heapSize=488 B 2023-03-20 10:59:22,182 INFO [RS:0;jenkins-hbase20:36041] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-03-20 10:59:22,182 DEBUG [RS:0;jenkins-hbase20:36041] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740, 39b9d351fb81939da79179eb526f8a01=hbase:namespace,,1679309780495.39b9d351fb81939da79179eb526f8a01.} 2023-03-20 10:59:22,183 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-03-20 10:59:22,183 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-03-20 10:59:22,183 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-03-20 10:59:22,183 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-03-20 10:59:22,183 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-03-20 10:59:22,183 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=20.94 KB heapSize=36.31 KB 2023-03-20 10:59:22,184 DEBUG [RS:0;jenkins-hbase20:36041] regionserver.HRegionServer(1505): Waiting on 1588230740, 39b9d351fb81939da79179eb526f8a01 2023-03-20 10:59:22,194 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=18.29 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/hbase/meta/1588230740/.tmp/info/20d919a1e2f64e4f84ca96563decf09f 2023-03-20 10:59:22,199 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 20d919a1e2f64e4f84ca96563decf09f 2023-03-20 10:59:22,266 INFO [regionserver/jenkins-hbase20:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-03-20 10:59:22,384 DEBUG [RS:0;jenkins-hbase20:36041] regionserver.HRegionServer(1505): Waiting on 1588230740, 39b9d351fb81939da79179eb526f8a01 2023-03-20 10:59:22,584 DEBUG [RS:0;jenkins-hbase20:36041] regionserver.HRegionServer(1505): Waiting on 1588230740, 39b9d351fb81939da79179eb526f8a01 2023-03-20 10:59:22,593 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/hbase/namespace/39b9d351fb81939da79179eb526f8a01/.tmp/info/2069f08be1b54a2ba2e9b2c381063d79 2023-03-20 10:59:22,601 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/hbase/namespace/39b9d351fb81939da79179eb526f8a01/.tmp/info/2069f08be1b54a2ba2e9b2c381063d79 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/hbase/namespace/39b9d351fb81939da79179eb526f8a01/info/2069f08be1b54a2ba2e9b2c381063d79 2023-03-20 10:59:22,608 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/hbase/namespace/39b9d351fb81939da79179eb526f8a01/info/2069f08be1b54a2ba2e9b2c381063d79, entries=2, sequenceid=6, filesize=4.8 K 2023-03-20 10:59:22,610 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 39b9d351fb81939da79179eb526f8a01 in 429ms, sequenceid=6, compaction requested=false 2023-03-20 10:59:22,613 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=588 B at sequenceid=93 (bloomFilter=false), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/hbase/meta/1588230740/.tmp/rep_barrier/0fa05063799b46a6b71cd0c5bef9b132 2023-03-20 10:59:22,624 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 0fa05063799b46a6b71cd0c5bef9b132 2023-03-20 10:59:22,666 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/hbase/namespace/39b9d351fb81939da79179eb526f8a01/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-03-20 10:59:22,667 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1679309780495.39b9d351fb81939da79179eb526f8a01. 2023-03-20 10:59:22,668 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 39b9d351fb81939da79179eb526f8a01: 2023-03-20 10:59:22,668 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1679309780495.39b9d351fb81939da79179eb526f8a01. 2023-03-20 10:59:22,786 DEBUG [RS:0;jenkins-hbase20:36041] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-03-20 10:59:22,988 DEBUG [RS:0;jenkins-hbase20:36041] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-03-20 10:59:23,060 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.08 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/hbase/meta/1588230740/.tmp/table/d86a77257d1f4a65a16b85679bee9a53 2023-03-20 10:59:23,064 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for d86a77257d1f4a65a16b85679bee9a53 2023-03-20 10:59:23,066 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/hbase/meta/1588230740/.tmp/info/20d919a1e2f64e4f84ca96563decf09f as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/hbase/meta/1588230740/info/20d919a1e2f64e4f84ca96563decf09f 2023-03-20 10:59:23,070 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 20d919a1e2f64e4f84ca96563decf09f 2023-03-20 10:59:23,070 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/hbase/meta/1588230740/info/20d919a1e2f64e4f84ca96563decf09f, entries=22, sequenceid=93, filesize=7.1 K 2023-03-20 10:59:23,072 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/hbase/meta/1588230740/.tmp/rep_barrier/0fa05063799b46a6b71cd0c5bef9b132 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/hbase/meta/1588230740/rep_barrier/0fa05063799b46a6b71cd0c5bef9b132 2023-03-20 10:59:23,078 INFO [regionserver/jenkins-hbase20:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-03-20 10:59:23,079 INFO [regionserver/jenkins-hbase20:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-03-20 10:59:23,080 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 0fa05063799b46a6b71cd0c5bef9b132 2023-03-20 10:59:23,081 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/hbase/meta/1588230740/rep_barrier/0fa05063799b46a6b71cd0c5bef9b132, entries=6, sequenceid=93, filesize=5.5 K 2023-03-20 10:59:23,081 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/hbase/meta/1588230740/.tmp/table/d86a77257d1f4a65a16b85679bee9a53 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/hbase/meta/1588230740/table/d86a77257d1f4a65a16b85679bee9a53 2023-03-20 10:59:23,085 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for d86a77257d1f4a65a16b85679bee9a53 2023-03-20 10:59:23,085 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/hbase/meta/1588230740/table/d86a77257d1f4a65a16b85679bee9a53, entries=9, sequenceid=93, filesize=5.5 K 2023-03-20 10:59:23,086 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.94 KB/21446, heapSize ~36.27 KB/37136, currentSize=0 B/0 for 1588230740 in 903ms, sequenceid=93, compaction requested=false 2023-03-20 10:59:23,111 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/data/hbase/meta/1588230740/recovered.edits/96.seqid, newMaxSeqId=96, maxSeqId=1 2023-03-20 10:59:23,113 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-03-20 10:59:23,115 INFO [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-03-20 10:59:23,115 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-03-20 10:59:23,115 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase20:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-03-20 10:59:23,188 INFO [RS:0;jenkins-hbase20:36041] regionserver.HRegionServer(1171): stopping server jenkins-hbase20.apache.org,36041,1679309777062; all regions closed. 2023-03-20 10:59:23,201 DEBUG [RS:0;jenkins-hbase20:36041] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/oldWALs 2023-03-20 10:59:23,201 INFO [RS:0;jenkins-hbase20:36041] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase20.apache.org%2C36041%2C1679309777062.meta:.meta(num 1679309780220) 2023-03-20 10:59:23,205 DEBUG [RS:0;jenkins-hbase20:36041] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/oldWALs 2023-03-20 10:59:23,205 INFO [RS:0;jenkins-hbase20:36041] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase20.apache.org%2C36041%2C1679309777062:(num 1679309779202) 2023-03-20 10:59:23,206 DEBUG [RS:0;jenkins-hbase20:36041] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 10:59:23,206 INFO [RS:0;jenkins-hbase20:36041] regionserver.LeaseManager(133): Closed leases 2023-03-20 10:59:23,206 INFO [RS:0;jenkins-hbase20:36041] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase20:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-03-20 10:59:23,206 INFO [regionserver/jenkins-hbase20:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-03-20 10:59:23,208 INFO [RS:0;jenkins-hbase20:36041] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:36041 2023-03-20 10:59:23,214 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-03-20 10:59:23,214 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase20.apache.org,36041,1679309777062 2023-03-20 10:59:23,214 ERROR [Listener at localhost.localdomain/43509-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@2bd94324 rejected from java.util.concurrent.ThreadPoolExecutor@2fe6c287[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2159] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-20 10:59:23,216 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-03-20 10:59:23,216 ERROR [Listener at localhost.localdomain/43509-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3ae504a5 rejected from java.util.concurrent.ThreadPoolExecutor@2fe6c287[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2159] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-20 10:59:23,217 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase20.apache.org,36041,1679309777062] 2023-03-20 10:59:23,218 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase20.apache.org,36041,1679309777062; numProcessing=1 2023-03-20 10:59:23,219 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase20.apache.org,36041,1679309777062 already deleted, retry=false 2023-03-20 10:59:23,219 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase20.apache.org,36041,1679309777062 expired; onlineServers=0 2023-03-20 10:59:23,219 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase20.apache.org,44257,1679309775907' ***** 2023-03-20 10:59:23,219 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-03-20 10:59:23,219 DEBUG [M:0;jenkins-hbase20:44257] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@93178a7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase20.apache.org/148.251.75.209:0 2023-03-20 10:59:23,219 INFO [M:0;jenkins-hbase20:44257] regionserver.HRegionServer(1145): stopping server jenkins-hbase20.apache.org,44257,1679309775907 2023-03-20 10:59:23,219 INFO [M:0;jenkins-hbase20:44257] regionserver.HRegionServer(1171): stopping server jenkins-hbase20.apache.org,44257,1679309775907; all regions closed. 2023-03-20 10:59:23,220 DEBUG [M:0;jenkins-hbase20:44257] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-20 10:59:23,220 DEBUG [M:0;jenkins-hbase20:44257] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-03-20 10:59:23,220 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-03-20 10:59:23,220 DEBUG [M:0;jenkins-hbase20:44257] cleaner.HFileCleaner(317): Stopping file delete threads 2023-03-20 10:59:23,220 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1679309778945] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.small.0-1679309778945,5,FailOnTimeoutGroup] 2023-03-20 10:59:23,220 INFO [M:0;jenkins-hbase20:44257] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-03-20 10:59:23,220 DEBUG [master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1679309778944] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase20:0:becomeActiveMaster-HFileCleaner.large.0-1679309778944,5,FailOnTimeoutGroup] 2023-03-20 10:59:23,220 INFO [M:0;jenkins-hbase20:44257] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-03-20 10:59:23,220 INFO [M:0;jenkins-hbase20:44257] hbase.ChoreService(369): Chore service for: master/jenkins-hbase20:0 had [] on shutdown 2023-03-20 10:59:23,220 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-03-20 10:59:23,220 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-20 10:59:23,221 DEBUG [M:0;jenkins-hbase20:44257] master.HMaster(1512): Stopping service threads 2023-03-20 10:59:23,221 INFO [M:0;jenkins-hbase20:44257] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-03-20 10:59:23,222 ERROR [M:0;jenkins-hbase20:44257] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-13,5,PEWorkerGroup] 2023-03-20 10:59:23,222 INFO [M:0;jenkins-hbase20:44257] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-03-20 10:59:23,222 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-03-20 10:59:23,224 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-03-20 10:59:23,224 DEBUG [M:0;jenkins-hbase20:44257] zookeeper.ZKUtil(398): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-03-20 10:59:23,224 WARN [M:0;jenkins-hbase20:44257] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-03-20 10:59:23,224 INFO [M:0;jenkins-hbase20:44257] assignment.AssignmentManager(315): Stopping assignment manager 2023-03-20 10:59:23,226 INFO [M:0;jenkins-hbase20:44257] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-03-20 10:59:23,227 DEBUG [M:0;jenkins-hbase20:44257] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-03-20 10:59:23,227 INFO [M:0;jenkins-hbase20:44257] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-20 10:59:23,227 DEBUG [M:0;jenkins-hbase20:44257] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-20 10:59:23,227 DEBUG [M:0;jenkins-hbase20:44257] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-03-20 10:59:23,227 DEBUG [M:0;jenkins-hbase20:44257] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-20 10:59:23,227 INFO [M:0;jenkins-hbase20:44257] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=345.17 KB heapSize=393.89 KB 2023-03-20 10:59:23,319 INFO [RS:0;jenkins-hbase20:36041] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase20.apache.org,36041,1679309777062; zookeeper connection closed. 2023-03-20 10:59:23,319 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-20 10:59:23,319 ERROR [Listener at localhost.localdomain/43509-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@422fb3bb rejected from java.util.concurrent.ThreadPoolExecutor@2fe6c287[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2159] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-20 10:59:23,319 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@6500bd04] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@6500bd04 2023-03-20 10:59:23,320 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): regionserver:36041-0x1002e5fb8b80001, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-20 10:59:23,320 ERROR [Listener at localhost.localdomain/43509-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@18194b80 rejected from java.util.concurrent.ThreadPoolExecutor@2fe6c287[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2159] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-20 10:59:23,321 INFO [Listener at localhost.localdomain/43509] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-03-20 10:59:23,660 INFO [M:0;jenkins-hbase20:44257] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=345.17 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/d3b4569af1e44381962ab60544efd147 2023-03-20 10:59:23,664 INFO [M:0;jenkins-hbase20:44257] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for d3b4569af1e44381962ab60544efd147 2023-03-20 10:59:23,665 DEBUG [M:0;jenkins-hbase20:44257] regionserver.HRegionFileSystem(485): Committing hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/d3b4569af1e44381962ab60544efd147 as hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/d3b4569af1e44381962ab60544efd147 2023-03-20 10:59:23,668 INFO [M:0;jenkins-hbase20:44257] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for d3b4569af1e44381962ab60544efd147 2023-03-20 10:59:23,669 INFO [M:0;jenkins-hbase20:44257] regionserver.HStore(1080): Added hdfs://localhost.localdomain:41729/user/jenkins/test-data/fba57c86-43c9-38e9-0a1f-37aba0d7538a/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/d3b4569af1e44381962ab60544efd147, entries=65, sequenceid=578, filesize=24.7 K 2023-03-20 10:59:23,669 INFO [M:0;jenkins-hbase20:44257] regionserver.HRegion(2948): Finished flush of dataSize ~345.17 KB/353458, heapSize ~393.88 KB/403328, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 442ms, sequenceid=578, compaction requested=false 2023-03-20 10:59:23,671 INFO [M:0;jenkins-hbase20:44257] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-20 10:59:23,671 DEBUG [M:0;jenkins-hbase20:44257] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-03-20 10:59:23,674 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-03-20 10:59:23,674 INFO [M:0;jenkins-hbase20:44257] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-03-20 10:59:23,675 INFO [M:0;jenkins-hbase20:44257] ipc.NettyRpcServer(158): Stopping server on /148.251.75.209:44257 2023-03-20 10:59:23,676 DEBUG [M:0;jenkins-hbase20:44257] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase20.apache.org,44257,1679309775907 already deleted, retry=false 2023-03-20 10:59:23,778 INFO [M:0;jenkins-hbase20:44257] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase20.apache.org,44257,1679309775907; zookeeper connection closed. 2023-03-20 10:59:23,778 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-20 10:59:23,778 ERROR [Listener at localhost.localdomain/43509-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@36b13017 rejected from java.util.concurrent.ThreadPoolExecutor@5eb95f6[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3600] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-20 10:59:23,778 DEBUG [Listener at localhost.localdomain/43509-EventThread] zookeeper.ZKWatcher(600): master:44257-0x1002e5fb8b80000, quorum=127.0.0.1:60226, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-20 10:59:23,778 ERROR [Listener at localhost.localdomain/43509-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@787c1001 rejected from java.util.concurrent.ThreadPoolExecutor@5eb95f6[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3600] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-20 10:59:23,779 WARN [Listener at localhost.localdomain/43509] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-03-20 10:59:23,783 INFO [Listener at localhost.localdomain/43509] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-03-20 10:59:23,892 WARN [BP-1547634163-148.251.75.209-1679309772220 heartbeating to localhost.localdomain/127.0.0.1:41729] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-03-20 10:59:23,892 WARN [BP-1547634163-148.251.75.209-1679309772220 heartbeating to localhost.localdomain/127.0.0.1:41729] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1547634163-148.251.75.209-1679309772220 (Datanode Uuid 20371a92-d140-4bdf-8ccd-77397eedee72) service to localhost.localdomain/127.0.0.1:41729 2023-03-20 10:59:23,896 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b3ae73f5-6ee4-322c-39f8-fb9a5f1d83f7/cluster_88c58e92-ae5f-fec6-f36b-653ddacb171c/dfs/data/data1/current/BP-1547634163-148.251.75.209-1679309772220] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-03-20 10:59:23,897 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b3ae73f5-6ee4-322c-39f8-fb9a5f1d83f7/cluster_88c58e92-ae5f-fec6-f36b-653ddacb171c/dfs/data/data2/current/BP-1547634163-148.251.75.209-1679309772220] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-03-20 10:59:23,936 INFO [Listener at localhost.localdomain/43509] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost.localdomain:0 2023-03-20 10:59:24,109 INFO [Listener at localhost.localdomain/43509] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-03-20 10:59:24,314 INFO [Listener at localhost.localdomain/43509] hbase.HBaseTestingUtility(1293): Minicluster is down